PaperlayerView exact revision on arXiv

Source-linked AI summary

FireRed-Image-Edit-1.0 Technical Report

Super Intelligence Team, Changhao Qiao, Chao Hui, Chen Li, Cunzheng Wang, Dejia Song, Jiale Zhang, Jing Li, Qiang Xiang, Runqi Wang, Shuang Sun, Wei Zhu, Xu Tang, Yao Hu, Yibo Chen, Yuhao Huang, Yuxuan Duan, Zhiyi Chen, Ziyuan Guo

arXiv:2602.13344v2cs.CVeess.IV

TL;DR

Practical image-editing evaluation can be disconnected from user experience, and available systems face challenges around accessibility and reproducibility. FireRed-Image-Edit combines data, training, and evaluation design, and reports state-of-the-art performance among open-source models and competitive results against proprietary systems.

  • Problem

    Conventional metrics can miss nuances of production-ready applications, creating a disconnect between academic scores and user experience.

    Source: §1 · p. 4

  • Method

    FireRed-Image-Edit is an end-to-end framework combining data curation, training methodology, and evaluation, including the REDEdit-Bench benchmark.

    Source: §1 · p. 4, §6 · p. 30

  • Results

    Experiments on REDEdit-Bench, ImgEdit, and GEdit report state-of-the-art performance among open-source models and competitive results against proprietary systems.

    Source: §6 · p. 30

  • Takeaways & Limitations

    The reported results indicate that carefully engineered efficiency and system-level optimization can rival brute-force scaling.

    Source: §6 · p. 30

Abstract

from arXiv · show

We present FireRed-Image-Edit, a diffusion transformer for instruction-based image editing that achieves state-of-the-art performance through systematic optimization of data curation, training methodology, and evaluation design. We construct a 1.6B-sample training corpus, comprising 900M text-to-image and 700M image editing pairs from diverse sources. After rigorous cleaning, stratification, auto-labeling, and two-stage filtering, we retain over 100M high-quality samples balanced between generation and editing, ensuring strong semantic coverage and instruction alignment. Our multi-stage training pipeline progressively builds editing capability via pre-training, supervised fine-tuning, and reinforcement learning. To improve data efficiency, we introduce a Multi-Condition Aware Bucket Sampler for variable-resolution batching and Stochastic Instruction Alignment with dynamic prompt re-indexing. To stabilize optimization and enhance controllability, we propose Asymmetric Gradient Optimization for DPO, DiffusionNFT with layout-aware OCR rewards for text editing, and a differentiable Consistency Loss for identity preservation. We further establish REDEdit-Bench, a comprehensive benchmark spanning 15 editing categories, including newly introduced beautification and low-level enhancement tasks. Extensive experiments on REDEdit-Bench and public benchmarks (ImgEdit and GEdit) demonstrate competitive or superior performance against both open-source and proprietary systems. To support future research, our code, models, and benchmark suite are publicly available at https://github.com/FireRedTeam/FireRed-Image-Edit/ .

1. Introduction

FireRed-Image-Edit presents an end-to-end framework that optimizes the model lifecycle, addressing gaps in efficient image-editing data construction and rigorous, standardized evaluation. It combines large-scale training with efficiency and optimization methods and introduces a practical, multidimensional editing benchmark.

1. Introduction

  • The introduction identifies inefficient high-quality editing-data construction and a lack of rigorous standardized benchmarks as gaps amid costly model scaling and opaque commercial systems.It also notes that synthetic data distilled from proprietary models has become a workaround for resource-limited academic research.Source: §1 · p. 4
  • FireRed-Image-Edit is a holistic end-to-end framework that optimizes data curation, architecture design, training efficiency, and inference optimization.The paper presents this lifecycle-wide optimization as the basis for its performance with large amounts of high-quality data.Source: §1 · p. 4
  • Approximately 100 million diverse image-text pairs support generation and editing, alongside condition-aware bucketing and stochastic reference-image dropout with prompt re-indexing.The sampler reduces padding waste while preserving spatial layout; dropout and dynamic re-indexing maintain semantic correspondence across randomized reference-image orderings. Feature caching, FSDP, gradient checkpointing, and mixed-precision training further increase throughput while preserving stability.Source: §1 · p. 4
  • Three coordinated mechanisms refine convergence: distributed stratified timestep sampling, logit-normal loss weighting, and exponential moving average of model weights.Synchronized rotation across GPU ranks aims for uniform noise coverage; loss weighting emphasizes intermediate timesteps, while averaging reduces transient fluctuations and supports robustness to distribution shifts.Source: §1 · p. 4
  • The benchmark evaluates editing from subtle texture changes to structural alterations across instruction alignment, background preservation, and photographic fidelity.It is designed to bring evaluation closer to production use and practical user experience.Source: §1 · p. 4

2. Data

FireRed-Image-Edit’s data pipeline combines large-scale, balanced generation and editing data with layered curation, synthetic data production, and instruction-focused annotation. Its design aims to provide broad task coverage and high-fidelity supervision for training.

2. Data

  • The corpus begins with 1.6 billion samples and retains 100M+ high-quality examples, evenly split between text-to-image and image-editing data.The original pool contains 900 million text-to-image pairs and 700 million editing pairs; cleaning, stratification, labeling, and two-stage filtering produce the retained set.Source: §2 · p. 4

2.1. Data Distribution

  • The retained dataset is approximately balanced between T2I and I2I, spanning nature, people, design, semantic editing, stylistic editing, and structural editing.T2I data preserves generative priors and supports semantic understanding; the six categories cover generation contexts and edits to image content, style, and structure.Source: §2.1 · pp. 5–6, Figure 3 · p. 5

2.2.1. Deduplication

  • Deduplication removes global near-duplicates, trivial or repeated I2I pairs, and subtle edit variants through progressively stricter comparisons.The pipeline uses shared-feature retrieval [25], CLIP [27] and internal embeddings for pair comparisons, then PSNR, SSIM, ISC21 [6], and CLIP similarity for fine-grained filtering.Source: §2.2.1 · pp. 6–7

2.2.2–2.2.5. Filtering

  • Additional filters remove statistically implausible image properties, visual artifacts, low-quality or unaesthetic samples, and detected AIGC content.Photometric descriptors screen exposure, color, entropy, texture, and sharpness; artifact checks include overlays and geometric misalignment, while IQA, aesthetics, and AIGC detectors address quality and synthetic-image biases.Source: §2.2.2 · p. 7, §2.2.3 · p. 7, §2.2.4 · p. 7

2.3. Data Production Engine

  • The data production engine expands editing coverage with instruction-driven experts and structured-control experts for precise, spatially sensitive edits.Instruction templates and an edit-target lexicon use VLM discovery, task inventories, and metadata for grounding; structured controls use signals such as SAM masks [3] and DWpose keypoints [46].Source: §2.3 · pp. 7–8

2.3. Data Production Engine; 2.4. Long-tail Data Supplementation

  • Model-free templates, task inversion, task splitting, and targeted retrieval supplement the production pipeline with deterministic edits and data for underrepresented tasks.Templates cover 3D transformations, structured layouts, and low-level enhancements; inversion swaps source and target images, while splitting breaks complex requests into atomic edits. For data gaps, retrieved imagery is passed through the engine to create new pairs.Source: §2.3 · pp. 8–9, §2.4 · p. 9

2.5. Captioning Engine; 2.5.1. Structured Captioning; 2.5.2. Instruction Captioning

  • A progressive captioning engine supplies structured image descriptions and varied editing instructions, from detailed change descriptions to concise and conversational requests.Detailed captions emphasize accurate differential reasoning; concise refinement varies wording while preserving intent, and user-like refinement adds natural, help-seeking phrasing. Mixing these with detailed descriptions trains across instruction lengths.Source: §2.5 · p. 9, §2.5.1 · p. 9, §2.5.2 · p. 10

2.6. Data Post-Filtering

  • A Qwen3-VL-8B evaluation model, trained on expert-annotated positive and negative examples, filters more than 100 million samples for instruction alignment and perceptual quality.The 100,000-example annotation set comprises 50,000 positive triplets and 50,000 instruction-perturbed negatives; filtering removes misaligned or low-fidelity samples using alignment and quality scores.Source: §2.6 · pp. 10–11

3. Model Training · 3.1. Architecture · 3.2. Training Efficiency Optimization

The training design couples a multimodal diffusion-transformer architecture with batching and instruction-alignment methods for variable-resolution, multi-reference editing. It also uses progressive training stages and system-level memory and throughput optimizations.

3. Model Training

  • Figure 7’s pipeline combines task- and aspect-ratio bucketing and prompt shuffle/drop with VAE-encoded images, Qwen VL conditions, MMDiT blocks, and shared-encoder RoI consistency loss.The loss compares predicted and ground-truth regions of interest to minimize identity discrepancy.Source: Figure 7 · p. 11

3.1. Architecture

  • Built on the open-source multimodal text-to-image foundation [38], the model scales training to approximately one billion image-text pairs across generation and instruction-based editing tasks.The corpus includes text-to-image, single- and multi-image-to-image generation, and complex instruction-based editing.Source: §3.1 · p. 11
  • The Double-Stream MM-DiT concatenates [38] text embeddings, VAE latent tokens, and editing reference-image features into a unified stream for dense bidirectional multimodal interaction.This setup is designed to preserve structural integrity while making precise modifications.Source: §3.1 · p. 11
  • 3D Unified RoPE gives reference and target image tokens shared spatial coordinates but distinct temporal intervals, alongside separate time-conditioning for clean references and noisy targets.The passage attributes stability and stylistic consistency in complex manipulations to this coordinate alignment and time-conditioning.Source: §3.1 · p. 12

3.2. Training Efficiency Optimization

  • The Multi-Condition Aware Bucket Sampler groups batches by aspect ratio and input-image count, constraining visual sequence length to device capacity to reduce GPU idle time.The sampler accounts for the variable number of input images in editing instructions and balances computational loads across GPUs.Source: §3.2 · p. 12
  • Bucket dimensions minimize aggregate cropping while keeping synchronized distributed batches tensor-consistent, reducing padding waste and preserving aspect-ratio information.The optimization selects among predefined resolution buckets for the images in a batch.Source: §3.2 · p. 12
  • Stochastic Instruction Alignment randomly drops and permutes reference images, then re-indexes prompt references to match their new order and preserve visual-linguistic alignment.This augmentation is intended to prevent overfitting to specific reference patterns and improve generalization in multi-reference editing.Source: §3.2 · p. 12
  • Training progresses through pretraining, continued pretraining, supervised fine-tuning, DPO, and DiffusionNFT; offline VLM embeddings, FSDP, checkpointing, BF16, and HSDP reduce overhead and improve throughput.FSDP sharding changes optimizer-state and gradient memory complexity from O(M) to O(M/N); HSDP confines sharding within nodes to reduce inter-node synchronization.Source: §3.2 · p. 12, Figure 8 · p. 13

3.3. Pre-training

Pre-training builds visual vocabulary and world knowledge by prioritizing data scale and semantic diversity over aesthetic quality. It combines flexible-resolution training, broad web data with hybrid captions, and a progressive noise-timestep curriculum.

Scalable Framework with Arbitrary-Resolution Adaptation

  • Dynamic aspect-ratio buckets enable native arbitrary-resolution training, preserve image composition, reduce padding, and alternate between text-to-image and single-image input modes.The two modes use paired image-text data for semantic alignment and unpaired images for visual feature learning.Source: §3.3 · p. 13

Massive-Scale Data Integration with Hybrid Conditioning

  • The model co-trains on broadly filtered web data using original metadata and structured synthetic captions to learn diverse world knowledge and improve prompt following.The inclusive filtering protocol retains effective samples rather than enforcing strict aesthetic curation or artificial class balancing.Source: §3.3 · p. 13

Curriculum Learning via Progressive Timestep Sampling

  • Training begins with high-noise timestep sampling to learn robust global structures, then shifts toward uniform sampling to refine details and textures while stabilizing learning from noisy web data.The curriculum transitions from coarse semantic structures and layouts to high-frequency generation as training converges.Source: §3.3 · p. 13

3.4. Continued Pre-training (CT)

Continued pre-training broadens task, resolution, and semantic coverage through unified sampling, dense-caption training, synthetic data, and semantic-cluster balancing.

Unified Task Sampling and Resolution Adaptability

  • CT jointly samples Text-to-Image, single-image, and multi-image tasks to strengthen the model across diverse input modalities.The unified strategy balances inputs across these generation paradigms during continued pre-training.Source: §3.4 · p. 13
  • Training spans nine aspect ratios, from 2:1 to 1:2, to limit overfitting to standard formats and preserve compositional integrity across canvas dimensions.The listed ratios are 2:1, 16:9, 3:2, 4:3, 1:1, 3:4, 2:3, 9:16, and 1:2.Source: §3.4 · p. 14

Synthetic Augmentation and Dense Semantic Alignment

  • CT combines curated Internet data with synthetic data for underrepresented domains and uses dense captions to align visual features with detailed textual cues.Dense-caption training is intended to improve understanding of long-tail vocabulary, rare objects, nuanced textures, and intricate scenarios.Source: §3.4 · p. 14

Cluster-Based Distribution Balancing

  • Semantic-cluster sampling balances category exposure instead of following raw dataset frequency, mitigating mode collapse and overfitting to dominant classes.The protocol applies uniform sampling across semantic clusters, including common objects and niche artistic styles.Source: §3.4 · p. 14

3.5. Supervised Fine-Tuning (SFT)

SFT uses high-resolution, carefully filtered data and structured instruction supervision to improve alignment and aesthetic fidelity while preserving semantic diversity. A smaller learning rate and EMA refine details and stabilize the final model.

3.5. Supervised Fine-Tuning (SFT)

  • SFT combines human-filtered 1024×1024 imagery with instruction-following captions and structured prompts to improve text–image alignment and textural quality.Dynamic resampling balances semantic categories to preserve long-tail concepts and pre-trained semantic diversity.Source: §3.5 · p. 14
  • A smaller learning rate refines high-frequency details and photorealistic attributes while protecting the global structure learned during pre-training.EMA averages model parameters across training steps to smooth optimization, mitigate checkpoint-specific biases, and support stability, aesthetic robustness, and generalization.Source: §3.5 · pp. 14–15

3.6. Reinforcement Learning with Human Feedback (RLHF)

The RLHF stage combines DPO with positive-sample reinforcement and online DiffusionNFT. It uses synthesized preference data, continuous reward signals, layout-aware OCR rewards, and semi-hard sample mining to support instruction alignment and editing quality.

3.6.1. Direct Preference Optimization (DPO)

  • To address DPO’s synchronous rise in Win Diff and Lose Diff, Asymmetric Gradient Optimization uses Positive Sample Reinforcement and SFT regularization to prioritize high-quality reconstruction.Setting ω > 1 amplifies the Win Diff gradient so alignment is driven primarily by high-fidelity chosen samples.Source: §3.6.1 · p. 15
  • The automated DPO data pipeline synthesizes instructions from atomic to complex tasks and mixes positive samples from expert and aesthetic-optimized models with VLM-filtered base-model negatives.This Mix-Policy construction aims to avoid self-sampling’s capability ceiling and improve visual fidelity and texture quality.Source: §3.6.1 · p. 15

3.6.2. Diffusion NFT

  • DiffusionNFT performs online reinforcement learning on the forward process using sampled-image optimality probabilities and weighted flow-matching error, contrasting positive and negative policies to favor high-reward regions.Unlike DPO, DiffusionNFT does not require paired data; its composite reward uses Fine-grained VLM and Layout-Aware OCR models, with semi-hard sample mining for data efficiency.Source: §3.6.2 · p. 15
  • VLM-based soft scores average numeric-rating-token expectations across stochastic inference passes, smoothing the reward landscape into a continuous reinforcement signal.The VLM generates Chain-of-Thought reasoning before scoring, and the scores are averaged across K inference passes.Source: §3.6.2 · pp. 15–16
  • For text editing, a layout-aware OCR reward evaluates character correctness, position, and scale, mitigating reward hacking and text collapse while producing more stable glyphs and natural typography.A gating mechanism activates layout terms only when text content is reasonably correct.Source: §3.6.2 · p. 16
  • Semi-hard mining selects edits with satisfactory mean rewards but unstable lower-bound scores, targeting the model’s capability boundary rather than trivial or collapse-prone samples.Hardness is estimated offline using the preceding DPO checkpoint and diverse candidate edits; the strategy aims to improve consistency and fidelity.Source: §3.6.2 · p. 16

3.7. Consistency Loss

The section introduces an identity consistency constraint to address identity drift in image editing, where pixel-level reconstruction losses may miss high-level semantic identity. It uses noise-dependent weighting and aligned face representations to guide identity preservation.

3.7. Consistency Loss

  • To address identity drift that pixel-level reconstruction losses can miss, the method tapers identity-loss weight as denoising shifts from high-noise structural formation to low-noise refinement.The quadratic decay is intended to reduce competition with fine-detail synthesis and avoid unintended artifacts in the low-noise regime.Source: §3.7 · pp. 16–17
  • The identity loss compares aligned regions from a one-step denoised estimate and ground-truth face, using a shared landmark-derived transformation and a pretrained semantic backbone.The backbone is described as robust to blur or artifacts in higher-noise estimates, providing guidance even when reconstruction is coarse.Source: §3.7 · p. 17
  • For images with multiple people, the framework aligns each detected face independently and averages cosine distance across subjects using a pretrained face-recognition backbone.The authors note that the alignment approach could potentially extend to other universal objects with suitable semantic encoders.Source: §3.7 · p. 17

3.8. Training Strategy.

The training strategy combines distributed timestep sampling, noise-level loss weighting, and model-weight averaging to support stable optimization and robust generalization.

Distributed Stratified Timestep Sampling

  • Stratified timestep sampling partitions T into K rank-specific intervals and periodically rotates assignments using a broadcast permutation seed, yielding globally uniform timestep coverage.Each rank samples uniformly from its assigned interval at each iteration.Source: §3.8 · p. 17

Logit-Normal Loss Weighting

  • Logit-Normal weighting suppresses gradient contributions from high- and low-noise extremes while concentrating capacity on intermediate diffusion timesteps.The weighting distribution is centered on the meaningful middle trajectory, where semantic structure and texture are established.Source: §3.8 · p. 17

Model Weight Averaging

  • EMA averages model parameters across converging iterations rather than relying on one final checkpoint, smoothing transient fluctuations and improving robustness to distribution shifts.The report describes this as an ensemble-in-weights approach.Source: §3.8 · pp. 17–18

4. REDEdit-Bench

REDEdit-Bench is a bilingual benchmark of 1,673 edit pairs across 15 categories, designed to assess instruction-following and preservation. Its evaluation pipeline reports that FireRed-Image-Edit achieves state-of-the-art performance among open-source models and adds text-editing metrics for character accuracy and visual fidelity.

4. REDEdit-Bench

  • REDEdit-Bench contains 1,673 bilingual Chinese–English edit pairs across 15 categories, with a balanced task distribution; Table 2 identifies it as the largest-scale benchmark and the only one covering all listed attributes.Source: §4 · p. 18, Table 2 · p. 18, Figure 9 · p. 18

4.1. Benchmark Construction

  • The benchmark pairs were built from over 3,000 real-world images, with professional staff writing instructions and multiple experts reviewing diversity and image-source legality.Source: §4.1 · p. 18

4.2. Evaluation Pipeline

  • FireRed-Image-Edit achieves SOTA performance among open-source models on REDEdit-Bench, evaluated primarily for Prompt Compliance, Visual Naturalness, and Physical & Detail Coherence.Gemini 3 Flash is used as the automated evaluator.Source: §4.2 · p. 18
  • Text editing is evaluated with OCR measures of character precision and a VLM Judge assessing SuccessEdit, OverEdit, Style, and background-fusion Consistency.OCR compares recognized text with ground truth using Levenshtein distance, completion rate, and word accuracy to produce a weighted normalized score; the VLM Judge complements spelling accuracy with visual-quality assessment.Source: §4.2 · p. 18

5. Performance and Evaluation

FireRed-Image-Edit is evaluated through blind human assessment and benchmark testing spanning general, text-centric, creative, and virtual try-on editing. The results report strong performance in instruction following, preservation, and visual quality across these settings.

5.1.1. Evaluation Dimensions

  • Annotators assess Prompt Following and Consistency Preservation through simultaneous, randomized, identity-blinded comparisons of multiple models’ outputs for the same input.Prompt Following measures execution of the intended edit and constraints; Consistency Preservation measures protection of non-edited content such as identity, background, structure, texture, and layout.Source: §5.1 · p. 19, §5.1.1 · p. 19

5.1.2. Results Analysis

  • In blind human evaluation, FireRed-Image-Edit leads on Prompt Following and Consistency, significantly outperforming open-source baselines and remaining competitive with commercial systems.It earns the highest Consistency Preservation score among compared methods.Source: §5.1.2 · p. 19, Figure 10 · p. 19

5.2.1. General Editing

  • FireRed-Image-Edit achieves state-of-the-art performance on ImgEdit and outperforms existing open-source models on GEdit.These benchmarks assess instruction completion or adherence and visual quality or fidelity.Source: §5.2.1 · pp. 19–20, Table 3 · p. 20, Table 4 · p. 20
  • FireRed-Image-Edit ranks highest among open-source models on REDEdit-Bench, emphasizing complex composite instructions and preservation constraints.The general-editing visualizations show object insertion and replacement, attribute changes, hybrid edits, and restoration with accurate localization and global consistency.Source: §5.2.1 · p. 20, Table 5 · p. 21, Table 6 · p. 21

5.2.2. Text-Centric Editing

  • On REDEdit-Bench’s text subset, FireRed-Image-Edit achieves leading results among open-source models in accurate text modification, style fidelity, and global coherence.The OCR and VLM Judge pipeline evaluates five dimensions, including character- and word-level accuracy, font-style preservation, visual consistency, and unnecessary modifications.Source: §5.2.2 · p. 21, Table 7 · p. 22

5.2.3. Creative Editing

  • Creative-editing examples show FireRed-Image-Edit translating high-level intents into coherent visual outcomes across structural abstraction, concept-driven synthesis, and imaginative edits.Examples include design sketches, sculptural styles, foldable designs, and floating objects.Source: §5.2.3 · p. 21

5.2.4. Try-on Editing

  • In virtual try-on visualizations, FireRed-Image-Edit produces more coherent clothing geometry, cleaner boundaries, and better alignment with textual styling instructions than existing baselines.The examples combine reference garments with constraints on fit, length, color, and accessories while preserving identity and realism.Source: §5.2.4 · p. 22

6. Conclusion

FireRed-Image-Edit is a diffusion transformer for instruction-based image editing, with contributions in data engineering, training methodology, and evaluation. Its conclusion highlights a large balanced training corpus, a multi-stage optimization framework, and benchmark results.

Data engineering

  • The curation pipeline distills over 100 million high-quality samples, evenly balanced between generation and editing, from approximately one billion multi-source image–text pairs.It integrates rigorous cleaning, fine-grained stratification, auto-labeling, and two-stage filtering.Source: §6 · p. 29

Training methodology

  • The efficiency-oriented training framework combines a multi-stage pipeline with methods for reducing padding overhead, improving instruction robustness, and stabilizing DPO optimization.The pipeline comprises pre-training, supervised fine-tuning, and reinforcement learning; its components include the Multi-Condition Aware Bucket Sampler, Stochastic Instruction Alignment, Asymmetric Gradient Optimization for DPO, Multi-Image DPO, Diffusion NFT, Consistency Loss, Distributed Stratified Timestep Sampling, Logit-Normal loss, and EMA.Source: §6 · pp. 29–30

Evaluation

  • Experiments on REDEdit-Bench, ImgEdit, and GEdit show state-of-the-art performance among open-source models and competitive results against proprietary systems.REDEdit-Bench spans 15 structured editing categories and contains 1,673 bilingual samples, including portrait beautification and low-resolution enhancement tasks.Source: §6 · p. 30

Annotation Prompt

The annotation prompt specifies a descriptive caption and structured visual analysis, returned in a JSON format covering image content, environment, and quality.

Step 1: Image Caption

  • Captions should describe objects, attributes, spatial relationships, and surroundings in natural language, quote visible text exactly, and avoid vague descriptions.The instructions ask for descriptive, unstructured language that is specific and accurate.Source: Annotation Prompt · p. 34

Step 2: Image Analysis

  • Image analysis classifies the image type and visual style and records detected watermarks and distracting anomalies such as QR codes or artifacts.Watermarks should be listed when present or marked “None.”Source: Annotation Prompt · p. 34

Output Format

  • The JSON output organizes results into caption, image type, style, anomalies, subjects, other elements, environment, and quality fields.Subject and environment fields capture appearance, clothing, action, emotion, setting, lighting, palette, and shot; quality includes resolution and watermarks.Source: Annotation Prompt · p. 34

Related papers

Loading 2602.13344v2…