PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching

Yoad Tewel, Yuval Atzmon, Gal Chechik, Lior Wolf

arXiv:2606.03911v1cs.CV

TL;DR

Visual editing training typically requires massive paired datasets, which are difficult to collect for scarce image edits and especially for video. ByG trains flow-matching editing models from unpaired data by combining frozen-model instruction cues, cycle consistency, and gradient routing. It reports over 75% user preference win rate against supervised baselines trained on millions of pairs and generalizes to unseen domains.

  • Problem

    Paired visual-editing examples are expensive and scarce for long-tail edits, limiting scalable training, especially for video.

  • Method

    ByG combines instruction-following cues from a frozen text-to-image model, cycle-consistency constraints, model unrolling, and gradient routing from clean predictions to noisy training states.

  • Results

    Over 75% user preference win rate was achieved against supervised baselines trained on millions of pairs across image and video editing, with generalization to unseen domains.

  • Takeaways & Limitations

    Unpaired flow-matching editing can match or outperform supervised baselines trained on millions of paired examples while generalizing to unseen domains.

  • Takeaways & Limitations

    The method inherits the pretrained base model’s knowledge and biases and performs more weakly on object removal because captions omit explicit absence.

Abstract

from arXiv · show

Modern generative models possess a deep understanding of visual content, yet training them for image editing typically requires massive datasets of paired examples. This limits scalability, especially for video editing where collecting paired data is prohibitively expensive. We propose Bootstrap Your Generator (ByG), a general framework for unpaired training of flow matching editing models. It leverages the base model's knowledge without any external signal. Our approach pairs instruction-following cues extracted from the frozen model with cycle-consistency for structure preservation. To make this tractable, we propose to route gradients from downstream losses over clean predictions to noisy training states. We demonstrate state-of-the-art results on challenging data-scarce image and video editing scenarios. Extensive evaluations and user studies show that our method effectively generalizes to unseen domains and outperforms supervised baselines trained on millions of samples. Analysis reveals that our gradient routing bridges the train-inference gap, and extracting semantic cues from a base model provides a robust training signal that obviates the need for external reward models.

1. Introduction

Visual editing methods largely depend on paired examples, but such data are scarce for long-tail image and video edits. ByG enables unpaired flow-matching editing by combining base-model instruction cues, cycle consistency, and gradient routing, achieving strong results across domains.

  • Motivation: Paired training data dominate visual editing, yet long-tail creative edits often lack collectable before-and-after examples.This limitation becomes more severe as editing expands to video and other modalities.
  • Approach: ByG uses the frozen base model for instruction-following supervision and cycle consistency to preserve source content using unpaired samples.The two signals target complementary goals: edit adherence and preservation of unchanged structure.
  • Approach: Model unrolling bootstraps training inputs, semantic cue extraction supplies instruction supervision, and gradient routing connects clean downstream losses to noisy training states.The gradient-routing mechanism is based on Straight-Through Estimation and addresses the train-inference gap.
  • Results: Over 75% user preference win rate was achieved against supervised baselines trained on millions of pairs across image and video editing.The method also outperforms supervised and zero-shot methods on long-tail style editing while generalizing to unseen styles.
  • Contributions: The framework is presented as the first general approach for unpaired training of flow-matching editing models.Its reported results include generalization across video and image domains and user-study evaluation.

2. Related Work

Prior visual-editing methods obtain training pairs through supervision, synthetic generation, simulation, or extraction, while unpaired and zero-shot approaches remain limited in scope or rely on external guidance. ByG differs by combining pretrained-model semantic directions, cycle consistency, and gradient routing for broader unpaired editing.

  • Image and Video Editing: Most image and video editing methods rely on large paired datasets, whose construction can introduce artifacts, realism gaps, narrow domains, or uncontrolled scene changes.Video methods particularly use key-frame editing, inpainting, and mixed image-video synthetic pipelines.
  • Gradient Routing: Gradient routing follows the Straight-Through Estimation pattern to provide clean continuous conditioning while backpropagating through the actual noisy prediction.This is described as mitigating exposure bias or the train-test gap in latent denoising.
  • Semantic Guidance: ByG extracts a semantic edit direction by contrasting frozen-model queries under target and source prompts on one state, rather than contrasting distinct image states.The method aligns this direction with cosine loss while cycle consistency preserves shared structure.
  • Cycle Consistency: CycleGAN-style cycle consistency enables unpaired translation, but prior cycle-based methods differ from ByG in reverse-pass conditioning, regularization, or data requirements.ByG maintains symmetric roles through gradient routing and uses directional regularization for semantic alignment.

3. Preliminaries - Supervised Image Editing

Flow-matching models learn generation by reversing a noising process, with supervised editing training using paired source and target images. The target is noised and the model learns its ground-truth velocity, but obtaining edit pairs at scale is expensive.

  • Flow Matching: Flow matching defines noisy samples as yt = (1 − t)y + tϵ, with t=0 representing clean data and t=1 representing pure noise.A velocity network reverses this process through multi-step integration from noise toward data.
  • Supervised Editing: In supervised image editing, the network receives a source image, editing instruction, and noised target image during training.The target image supplies the data needed to construct the noisy training input.
  • Supervised Editing: The supervised objective trains the model against the ground-truth velocity ut = ϵ − y.This velocity is defined from the sampled noise and clean target image.
  • Motivation: Obtaining edit pairs at scale is expensive, motivating an unpaired objective that bypasses paired-target supervision.Without paired data, both the noisy target input and ground-truth velocity are unavailable.

4. Method

ByG trains flow-matching editing models without paired edited targets by bootstrapping noisy inputs, using frozen text-to-image guidance and cycle consistency. Gradient routing lets clean predictions provide conditioning while losses train noisy states, addressing the train–inference mismatch.

  • Unpaired training: The framework adapts a pretrained text-to-image model into an editing model using only unpaired source and target captions, without ground-truth edited images.Training uses source images, source captions, target captions, and edit instructions.
  • Noisy input targets: A frozen EMA copy generates noisy pseudo-targets through multi-step sampling, allowing the model to bootstrap its own training inputs.The EMA copy is updated from the trainable model and stabilizes the bootstrapping loop.
  • Instruction following: The prior loss extracts instruction-following supervision from the frozen base model by aligning predictions with the difference between target-caption and source-caption velocities.Directional alignment is supplemented with an MSE term that regulates velocity magnitude.
  • Source preservation: Cycle consistency preserves source content by editing the source forward and reconstructing it with the inverse instruction, while symmetric prior loss guides the reverse pass.The reverse model receives the predicted edit as a condition and must recover the original source.
  • Gradient routing: Gradient routing conditions the reverse pass on a clean EMA estimate but sends gradients through the one-step prediction, adapting Straight-Through Estimation to latent denoising.This prevents blurry one-step predictions from creating a train–inference mismatch and helps retain useful conditioning.
  • Source preservation: An identity loss directly trains preservation of condition information by requiring reconstruction when the source is supplied as both input and condition.The inverse instruction is already satisfied in this setting.

5. Experiments

Experiments evaluate ByG on video editing, long-tail style transfer, and general image editing using human preference, reference-based, and judge-based metrics. Across these settings, the method performs strongly without paired data, including on unseen styles and video domains.

  • The evaluation covers instruction-based image and video editing across long-tail and general-purpose benchmarks.
  • 5.1.1. VIDEO EDITING: In video editing, the user study compares ByG with Ditto, a supervised model trained on one million video editing pairs.Eight participants rated 238 comparisons after selecting the best of four samples per method.
  • 5.1.1. VIDEO EDITING: 75.3%±2.2% overall win rate shows that ByG outperforms the supervised baseline without paired data.Win rates are 70.0%±5.4% for cartoon targets and 80.5%±2.9% for photo-realistic targets.
  • 5.1.1. VIDEO EDITING: 85.0% versus 15.0% on out-of-distribution 3D-CGI inputs demonstrates strong transfer beyond the video training domains.The quantitative video evaluation also reports advantages in edit success, source preservation, and motion fidelity, while matching source videos on aesthetic quality and temporal flickering.
  • 5.1.2. LONG-TAIL STYLE EDITING: On six long-tail styles, ByG achieves the highest overall score in both photorealistic→style and style→photorealistic directions without style-specific training.The benchmark includes GTA V, Minecraft, American comic, low-poly 3D, voxel, and Lego styles; gains are particularly strong on Semantic Consistency.
  • 5.2. General Image Editing: On GEdit-Bench, ByG is competitive with FLUX-Kontext across most categories, outperforming it on motion, human-centric, and style changes.FLUX-Kontext remains stronger on subject removal and text editing, where precise paired examples provide an advantage.

6. Ablation Study

The ablations show that gradient routing, cycle consistency, directional regularization, bootstrapping, and regularization each address distinct editing failures. Their removal reveals trade-offs between edit strength, source preservation, stability, and instruction following.

  • Qualitative Results: The qualitative style-editing results show better target-style matching while preserving source content.The style-reference column is provided for viewing and is not used during training or evaluation.
  • Source Preservation: Removing gradient routing or cycle loss improves edit success but reduces source preservation.Gradient routing also addresses the train–test mismatch caused by noisy one-step conditioning.
  • Source Preservation: Removing the directional term increases drift from the source by pulling outputs more strongly toward the target instruction.The ablation retains only the prior’s MSE component.
  • Instruction Following: Removing regularization causes identity collapse: source preservation remains high, but edit success becomes very low.The model produces minimal change and preserves the source largely unchanged.
  • Gradient Routing: One-step predictions are blurry and lack fine details, whereas multi-step sampling produces clean outputs.Gradient routing conditions on the clean multi-step estimate while backpropagating through the one-step prediction.
  • Training Stability: Without bootstrapping, distribution mismatch destabilizes training and degrades both edit success and source preservation.The forward process is driven by a noised source image rather than a noised edited-image prediction, producing edit artifacts.

7. Limitations

The method inherits the pretrained base model’s knowledge and biases, limiting reliability in domains the base model does not understand. Caption-based supervision is also weaker for object removal because omission does not explicitly describe absence.

  • Scope: The method cannot reliably edit toward target domains that the pretrained base model does not understand.This limitation follows from inheriting the base model’s knowledge and biases.
  • Object Removal: Caption-based supervision is weaker for object removal because the target caption omits the object without explicitly describing its absence.For example, “a sofa” does not explicitly signal that the cat should be removed.
  • Object Removal: This weaker supervisory signal makes removal edits harder to learn than additive or transformative edits.

8. Conclusion

The paper presents unpaired visual editing by combining frozen text-to-image model cues with cycle consistency and gradient routing. Experiments report parity or improvements relative to supervised baselines while generalizing to unseen domains.

  • Conclusion: The framework trains image and video editing models without paired supervision by combining frozen text-to-image cues with cycle-consistency constraints.
  • Conclusion: Gradient routing bridges the train-inference gap by conditioning on clean predictions while backpropagating through noisy states.
  • Conclusion: Experiments show that the unpaired method matches or outperforms supervised baselines trained on millions of paired examples and generalizes to unseen domains.

Impact Statement

The paper frames its contribution as advancing machine learning through a framework for visual editing. Its stated broader-impact concern is potential misuse for misleading visual content, alongside a call for detection methods and responsible-use guidelines.

  • Impact Statement: The work aims to advance the field of machine learning through visual editing research.
  • Impact Statement: The method could potentially be misused to create misleading visual content.
  • Impact Statement: The authors encourage detection methods and responsible-use guidelines alongside editing capabilities.

B. Implementation Details

The implementation adapts pretrained image and video generators for editing by concatenating source and noisy latents, then fine-tuning with LoRA and the paper’s editing objectives. Video conditioning aligns clean source tokens with noisy input tokens, while qualitative comparisons assess realism, instruction following, and content preservation.

  • Architecture: Image editing conditions FLUX.1-dev by concatenating source-image VAE tokens with noisy target latents and fine-tunes the model using rank-64 LoRA.The source encoding is concatenated along the token sequence dimension.
  • Training: Image training uses a learning rate of 3e−4, batch size 8, 30000 steps, and identity-only optimization for the first 200 steps.The setup uses AdamW with weight decay 10−2 and λprior = 1.0, λid = 0.2, and λcycle = 1.0.
  • Computation: Training takes ∼3× longer per step than supervised training, but meaningful editing appears after as few as 1000 steps and inference adds no overhead.The reported per-step times are 2.9s versus 0.97s, with 20 inference steps.
  • Architecture Adaptation: Video editing adapts Wan2.2 by concatenating source-video and noisy input latents along the token dimension while leaving temporal attention unchanged.The same prior, cycle, and identity objectives are applied to video latents as to images.
  • Architecture Adaptation: Shared positional encodings align source and noisy video tokens, while their different noise levels help attention distinguish clean conditioning features from noisy inputs.The source video remains noise-free, whereas the input is noisy according to timestep t.
  • Video Training Setup: Video training uses rank-64 LoRA on 8 H100 GPUs for 750 steps at 320x576 resolution, with a 10% identity-loss probability.Inference uses UniPC at native 480x832 resolution.
  • Qualitative Comparisons: On GEdit-Bench, the method is often more realistic than FLUX-Kontext and follows instructions more faithfully than the zero-shot baseline.The comparison attributes possible Kontext artifacts to synthetic paired training data.

C.1. EMA Ablation in Video Editing

The EMA ablation evaluates pseudo-targets generated from an exponential moving average of training weights rather than a stop-gradient copy of current weights. EMA improves source preservation and motion fidelity, with a small editability cost.

  • EMA Ablation: EMA pseudo-targets use an exponential moving average of training weights instead of a stop-gradient copy of the current weights.The paper evaluates this as an alternative to its default non-EMA video configuration.
  • Results: EMA improves source preservation and motion fidelity while reducing editability slightly.The reported metrics are DINO Sim. for source preservation and CLIP dir for editability.
  • Evaluation Setting: The main-paper qualitative results and user study use the default non-EMA variant.The EMA variant is evaluated separately in the ablation.
  • Training Tuple Context: The ablation section follows the construction of unpaired training tuples containing source and target prompts, forward and reverse instructions, and a source image.These tuples define the inputs used by the unpaired objective.

D.1. Image Data

The data pipeline uses vision-language models to create structured image-editing specifications and applies constrained caption processing for style-diverse video data. Manual verification and benchmark construction support evaluations across image and video editing settings.

  • Image Data: A VLM converts each captioned image into edit type, source caption, instruction, target caption, and reverse instruction fields.The pipeline uses Qwen3-VL-30B-A3B-Thinking and retains examples only when required JSON fields are present.
  • Image Data: The edit taxonomy covers color, texture, shape, addition, removal, replacement, background, style, action or pose, and text manipulation.Prompts require one visually noticeable edit affecting a prominent object or the entire background.
  • Image Data: Prompt constraints make reverse instructions self-contained, target captions describe only final images, and stylized source captions explicitly include style.The pipeline occasionally forces non-photorealistic images into photorealistic style conversion with pstyle=0.15.
  • Video Data: Video captions are sampled from VideoUFO, sanitized to remove existing style hints, and wrapped with randomly selected cartoon or photo-realistic styles.Generated videos are manually verified and mismatches are discarded.
  • Benchmark Construction: The real-world video benchmark samples UltraVideo clips labeled cartoon, photo-realistic, or 3D-CGI, resized to 480×832 resolution with 81 frames.Style labels are assigned through manual inspection of a larger candidate pool.
  • User Study: The video user study collected 238 votes from 8 participants comparing two methods on instruction following and content preservation.Participants judged which edited video was better given a reference video and editing instruction.
  • Qualitative Evaluation: Additional video comparisons report better target-style matching while preserving content.The figure provides qualitative evidence for the video-editing evaluation.
Loading 2606.03911v1…