PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowMimic: Mask-free Visual Editing and Generation with Pixel-pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry

Dingyun Zhang, Lixue Gong, Wei Liu

arXiv:2607.18227v1cs.CV

TL;DR

Video-editing data collection is costly and task-dependent, while existing models often rely on external region guidance. FlowMimic generates video-editing samples from image-editing pairs with a pixel-pair temporal warped flow field, aligns image and video outputs through modality mimic losses, and internalizes editing-region localization; experiments show multi-task editing capability and relatively aligned modalities, subject to data-quality and diversity limits.

  • Problem

    Video-editing datasets rely on labor-intensive curation and error-prone synthesis, limiting scalability, while language-guided editing often depends on external masks or auxiliary localization modules.

  • Method

    FlowMimic generates video-editing samples online from image-editing samples using a pixel-pair temporal warped flow field, aligns modalities with mimic losses, and trains sense-related region-aware losses.

  • Results

    FlowMimic learns multi-category video editing from online-generated data, yields more camera-authentic T2I results than the pretrained T2IV model, and relatively aligns video- and image-editing output distributions.

  • Takeaways & Limitations

    Image-editing samples can support scalable video-editing training across diverse tasks, while sense losses reduce reliance on editing-region mask sequences.

  • Takeaways & Limitations

    Training is constrained by low-quality, low-frame-rate video data, contaminated or scarce image-editing samples, and limited task-specific data such as face and head swapping.

Abstract

from arXiv · show

In line with the prevailing direction of vision research, we explore the integration of both generation and editing capabilities for video and image modalities within a single model. Current approaches to collecting video editing data typically depend on labour-intensive, time-consuming curated procedures--involving object mask annotation, the use of error-introducing pair synthesis via I2V model and ControlNet-like guidance, and VLM-based quality filtering or refinement--and demonstrate limited task scalability. As a result, the diversity of editing tasks remains substantially narrower than that available for image editing models. We develop a pixel-pair temporal warped flow field that can directly generate corresponding video editing samples in real time from image editing samples, and we demonstrate across multiple levels of video editing tasks that a model can learn video editing using only such data. We regard the image modality as a particular form of the video modality. Accordingly, we design a modality mimic generation loss and a modality mimic editing loss to relatively align the capabilities--and thereby the output distributions--of the two modalities through mutual imitation. Moreover, language-based visual editing entails the comprehension of the editing instruction and the reference visual content, the localization of the region corresponding to that instruction within the reference visual contents, and the modification of that region alone. Existing approaches predominantly rely on external aids, such as fine-tuning an additional MLLM or explicitly supplying a mask sequence as auxiliary input during inference. In contrast, we aspire for the model to internalize this capability. To that end, we introduce sense-related tasks--for instance, referring expression segmentation--along with corresponding editing-region-aware latent-level loss and attention-level loss.

1. Introduction

FlowMimic proposes scalable video-editing data generation from image-editing samples by preserving pixel-level correspondences over time, while aligning image and video modalities and internalizing instruction-guided editing-region localization.

  • Motivation: Existing video-editing data pipelines require labor-intensive mask annotation, curated pairs, error-prone I2V synthesis, and manual or model-based filtering.These task-dependent procedures limit scalability and editing-task diversity relative to image-editing models.
  • Pixel-pair Temporal Warped Flow Field: FlowMimic generates video-editing samples in real time from existing image-editing samples alone using a pixel-pair temporal warped flow field.The approach treats image pairs as corresponding pixel sets and propagates their spatial relationships through time.
  • Pixel-pair Temporal Warped Flow Field: The method assumes temporal consistency of pixel-level editing correspondences is sufficient for learning, even when generated objects deform unnaturally in later frames.At inference, the model is intended to produce naturally coherent editing effects on real source videos.
  • Modality Mimicry: Image and video are treated as a unified modality through first-frame modality mimic generation and editing losses that align T2I with T2V and I2I with V2V.The editing mimic loss uses the easier I2I response to guide V2V behavior, while the generation loss encourages T2I to mimic T2V photorealism.
  • Results: FlowMimic learns editing across multiple categories, produces more camera-authentic T2I results than the pretrained T2IV model, and relatively aligns image- and video-editing output distributions.Cross-attention analysis further indicates reliable localization of the instructed editing region.
  • Sense Losses: Sense-related tasks and editing-region-aware latent- and attention-level losses teach the model to localize and modify only instruction-relevant regions without external masks or auxiliary modules.Referring expression segmentation provides a natural related task and readily available segmentation masks for training.

2. Related Work

Related work spans large-scale image-editing datasets, synthetic video-editing data pipelines, training-free and training-based video editing, and task-specific or unified editing models.

  • Extensive Image Editing Datasets: Image-editing datasets have expanded from manually annotated collections to large curated resources, including MagicBrush, Pico-Banana-400K, GPT-IMAGE-EDIT-1.5M, and ImgEdit.These datasets differ in scale, construction, content preservation, instruction faithfulness, and task complexity.
  • Video Editing Data: Synthetic video-editing datasets use approaches such as Prompt-to-Prompt adaptation, CLIP filtering, online video collection, VLM orchestration, I2V synthesis, and category-specific workflows.Rendered data from 3D game engines is another option for specific subtasks such as camera-controllable body animation.
  • Video Editing Paradigms: Training-free video editing commonly maps source videos into initial noise through inversion, making editing quality dependent on the quality of the inverted latent.This creates stringent requirements for the inversion method.
  • Video Editing Paradigms: Training-based methods fine-tune video models or add LoRA parameters, while other approaches use context adapters or propagate first-frame edits through videos.Examples include I2VEdit, GenProp, VACE, and VideoPainter.
  • Task-specific Editing Models: Task-specific models address relighting, background replacement, face swapping, and virtual try-on using specialized modules, guidance signals, or datasets.Examples include Light-A-Video, ANYPORTAL, VFace, and KeyTailor.
  • Unified Models: Unified editing research combines multiple tasks or image and video modalities with architectures such as interleaved-token transformers and datasets built from existing models and VLMs.EditVerse and InstructX represent this multi-task or multi-modal direction.

3. Methodology

FlowMimic unifies video and image generation and editing by treating images as single-frame videos, generating temporally aligned video-editing data from image pairs, and training modality-mimic and sense-aware objectives. The approach targets scalable video editing, cross-modal capability alignment, and internalized instruction-guided region editing.

  • Model and flow matching: Wan2.1 is a pretrained T2IV model built on DiT, using flow matching to learn velocity fields that transport noise toward target data.The model encodes video with a 3D spatio-temporal VAE, perturbs latent tokens with Gaussian noise, and predicts the target velocity during training.
  • Pixel-pair temporal warped flow field: A pixel-pair temporal warped flow field applies identical time-varying spatial deformations to source and target images, producing video pairs with persistent pixel-wise editing correspondences.The construction views image pairs as corresponding pixel sets and synchronously moves them across temporal planes.
  • Pixel-pair temporal warped flow field: Random-seed-controlled deformation parameters vary motion strength while preserving the same deformation sequence for each source-target pair.Gentle motion yields smooth inter-frame transitions, whereas stronger motion produces more pronounced temporal variation and affects convergence and editing effects.
  • Modality mimic losses: The model treats images as single-frame videos and aligns T2I with T2V output distributions through a first-frame modality mimic generation loss.The loss compares estimated original latents from image and video processing, steering the two modalities toward a unified generative distribution.
  • Modality mimic losses: A modality mimic editing loss makes V2V imitate faster-converging I2I behavior while making I2I imitate the higher-quality converged V2V behavior.This mutual imitation is intended to accelerate V2V convergence and improve I2I editing quality.
  • Sense losses: Sense-related tasks and editing-region-aware latent and attention losses train the model to understand instructions, localize target regions, and edit only those regions without external masks.The contour loss applies flow matching over the latent editing mask, while the cross-attention loss directly regularizes linguistic-visual alignment.

4. Experiments

Experiments evaluate FlowMimic’s online video-editing data paradigm, modality-mimic losses, sense-related tasks, and performance across classical, practical, stylized, and exploratory settings.

  • Modality Mimic Results: Compared with the pretrained T2IV model, FlowMimic improves cinematic realism in T2I generation under identical cinematic prompts.The modality mimic generation loss is intended to bring T2I generation closer to T2V cinematic realism.
  • Sense-related Tasks: Sense-related tasks and losses improve focus on visual regions corresponding to referring expressions compared with the pretrained T2IV model.
  • Classical Video Editing Tasks: Using only online-constructed video editing data, the model masters multi-task editing, while sense-related losses remove reliance on editing-region mask sequences.Multi-task training slows convergence but supports more comprehensive editing capability.
  • More Practical and Interesting Video Editing Tasks: The model handles a broad range of practical editing tasks and responds to multi-object and multi-task scenarios despite training data containing single-object, single-task samples.Additional tasks include image-to-normal-map, depth-map-to-image, skeleton-map-to-image, and sketch-to-image editing.
  • Further Exploration of Editing Capabilities: The model generalizes to unseen body-animation conditions and inference sequences extending beyond the training-frame distribution.The reported results indicate a degree of temporal generalization and generalization to animating reference images with condition maps.
  • Stylized T2V Results: Compared with the pretrained T2IV model, FlowMimic achieves better stylistic responsiveness for stylized T2V prompts.The comparison uses three common oil-painting styles with identical prompts and random seeds.

5. Conclusion

FlowMimic unifies image and video generation and editing through online pixel-wise temporal correspondence, modality mimic losses, and internalized region understanding. The paper reports broad editing generalization while identifying limitations in training-data quality, sense-task diversity, and task-weight tuning.

  • Conclusion: FlowMimic enables scalable, real-time generation of video editing data from image editing data alone through a pixel-wise temporal warped flow field.The paradigm is intended for freely defined editing tasks when source and target images have loose layout correspondence.
  • Conclusion: Modality mimic generation and editing losses align image and video output distributions by transferring strengths between the two modalities.The losses encourage T2I outputs toward T2V cinematic realism and steer video editing toward the image-editing distribution.
  • Conclusion: Sense-related tasks and region-aware latent- and attention-level losses target instruction comprehension, region localization, and modification of only the specified region.
  • Future Works: The image-data paradigm shows promise for generalizing to high-level video understanding, low-level video processing, and controllable video generation.The authors identify improving image-sample quality and quantity as a plausible path for further improvement.
  • Future Works: Sense-related data are limited in referring-expression variety and often use coarsely delineated instance masks.The stated improvement direction is to add broader categories and more finely detailed contours.
  • Future Works: Available video and image training data include low-quality footage, temporal smoothness degradation, scarce task samples, inconsistent swaps, and blurry inpainting results.
  • Future Works: The study does not perform dedicated hyperparameter tuning for individual task weights, although increasing weights can expedite convergence for prioritized tasks.

A. FFP Details and Color Block Abstraction

The color block abstraction method creates online video editing samples to test first-frame propagation under spatial drift and misalignment. It uses altered color blocks as surrogate objects and evaluates whether replaced objects acquire plausible motion.

  • FFP Details: First-frame propagation training uses mostly object addition, object removal, and outpainting, with limited stylization, object replacement, and background replacement data.All training data is generated online from single-reference image editing samples.
  • Color Block Abstraction: The method replicates an image into three copies and inserts random-color rectangles whose sizes and positions vary under categorized spatial relationships.The rectangles represent the original subject, its replacement, and altered propagation conditions.
  • Color Block Abstraction: Color block abstraction represents original and replacement objects with blocks differing in color, shape, and position to simulate localization drift.The method targets discrepancies between edited propagation frames and the original object in the first source frame.
  • FFP Details: The first and third images form the image pair, while the second image supplies the reference propagation first frame for online video-sample generation.This construction produces replaced-object motion consistent with the original object, as demonstrated in the reported results.
  • Color Block Abstraction: Variants include smoothly moving blocks, occlusion simulations for object removal, deforming shapes, and color instance masks.Although validated during exploration, the paper excludes this strategy from its reported results to focus on preserving pixel-wise editing effects without third-party editing errors.

B. The Modality Mimicry Paradigms

The paper proposes viewing modality mimicry as a general paradigm for transferring performance from samples with favourable conditions to otherwise matched samples.

  • The Modality Mimicry Paradigms: Modality mimicry compares two samples with identical configurations that differ in one favourable condition.The paradigm is presented as potentially generalizable beyond the concrete implementations discussed in the paper.

B.1. On-policy Self-distillation of Flow Matching

The paper interprets modality mimicry as a form of on-policy self-distillation and considers extending it to other modality and conditioning gaps.

  • On-policy Self-distillation of Flow Matching: For T2V and T2I, the favourable condition is temporal attention alongside subsequent frames, while other inputs remain matched.The passage identifies original latents, noises, timesteps, noisy latents, and caption embeddings as the other configurations.
  • On-policy Self-distillation of Flow Matching: The editing mimicry loss is proposed to encourage S2I capability to imitate the more photorealistic and lifelike S2V counterpart.The passage motivates this extension through an observed gap in cinematic realism and identity preservation between S2I and S2V.
  • On-policy Self-distillation of Flow Matching: Modality mimicry can be viewed as on-policy self-distillation for flow matching models, where a student generates mistakes and a same-origin teacher corrects them in real time.This interpretation is presented in the context of the FlowMimic-OPSD discussion.
  • On-policy Self-distillation of Flow Matching: Modality mimicry may also target favourable conditions such as more detailed captions, additional references, better-quality references, or views with larger visible portions.The stated aim is to enhance performance or capability associated with those conditions.

B.2. Modality Self-alignment of Multi-task Models

The paper generalizes modality mimicry into modality self-alignment for related multi-task models, arguing that joint training alone may not align their output distributions.

  • Modality Self-alignment of Multi-task Models: Modality self-alignment seeks aligned output distributions when related tasks have nearby inputs in shared input and output feature spaces.The framework is described for tasks that are isomorphic or inverse and share representational spaces.
  • Modality Self-alignment of Multi-task Models: Simply training multiple tasks together is reported as insufficient for aligning T2I and T2V outputs in some scenarios.The passage refers to results exemplified in Fig. 3 and motivates explicit mimicry constraints.
  • Modality Self-alignment of Multi-task Models: The proposed response is to use modality mimicry methods to explicitly constrain alignment between related task outputs.The passage presents this as a distillation- and distribution-alignment-oriented generalization.
  • Modality Self-alignment of Multi-task Models: For I2T and T2I, output distance can be computed in a shared CLIP-like bimodal feature space instead of between estimated original latents or perceptual feature scores.This example replaces the image-video modality pair with image-text modalities.

C.1. Measures for Modality Mimicry

The paper compares MSE with its adopted modality mimic generation loss, finding that MSE provides a less flexible signal and performs worse in modality mimicry.

  • Loss comparison: MSE was applied between flattened estimated original latents from T2V first-frame and T2I samples.The comparison uses F(ẑ^1_t2v-f,0) and F(ẑ^1_t2i,0).
  • Loss comparison: MSE behaves more like latent-space reconstruction than distribution alignment, reducing flexibility during modality-mimic learning.The adopted loss instead encourages learning distributions of abstract visual perceptual characteristics.
  • Empirical comparison: MSE produced slower convergence and inferior modality-mimic performance relative to the adopted generation loss.The difference was observed through visual perceptual discrepancies between T2I outputs and first-frame long-clip T2V results.
  • Empirical comparison: For modality mimic editing, MSE occasionally caused abnormal global color-flickering artifacts in V2V outputs.

C.2. Mimicry on Visual Perceptual Feature Scores

The paper evaluates modality mimicry through visual perceptual feature and cinematic-aesthetic scores, but finds these signals weaker overall than the adopted latent-distribution loss.

  • Feature-score design: The feature-score approach extracts visual perceptual features from decoded estimated original latents and compares corresponding T2V and T2I features.A cosine-similarity loss is given as an example of the comparison.
  • Feature-score design: Cinematic-aesthetic scores are computed with an internal feature-score model, using an L1 loss only when the T2I score exceeds the T2V score.The score loss is zero otherwise, limiting effects from possible score-estimation errors.
  • Empirical comparison: Cinematic aesthetics improved for T2I outputs, but convergence speed and overall performance were markedly weaker than with the adopted modality mimic generation loss.The latent-based loss supplies a more direct signal and is not bottlenecked by a third-party feature extractor.
  • Empirical comparison: HPSv2-based visual perceptual feature scoring performed worse than the cinematic-aesthetic feature estimator.The experiment used low-noise timestep sampling within [80, 390).
  • Possible extensions: Score-vector distributions could alternatively support an auxiliary distributional loss after softmax transformation.The passage also identifies KL divergence as a possible measure when image score-distribution estimators are available.

C.3. Mimicry on Attention Features

The paper tests self-attention key and value features as a modality-mimic medium, but this attention-only loss underperforms the adopted loss.

  • Attention-feature design: The experiment extracts self-attention key and value features from the last DiT block for T2V first-frame and T2I estimated latents.
  • Attention-feature design: The attention-feature modality mimic loss minimizes squared differences between corresponding T2V and T2I key and value features.The formulation contains separate K and V terms.
  • Empirical comparison: The attention-only loss showed a noticeable convergence and performance gap relative to the original adopted modality mimic generation loss.The paper retains it as a possible auxiliary loss but does not use it for simplicity.

C.4. Loss-free Modality Mimicry

The paper investigates loss-free modality mimicry by swapping T2V and T2I estimated latents, then relying on ordinary flow-matching losses rather than a dedicated mimicry loss.

  • Latent swapping: Loss-free generation mimicry swaps the first-frame estimated original latent of a T2V sample with the corresponding T2I latent.
  • Latent swapping: The procedure also exchanges the corresponding estimated noise latents between the T2V and T2I samples.
  • Training objective: The swapped samples contribute only their respective flow-matching losses, without an explicit modality mimic generation loss.The flow-matching loss remains tied to the estimated initial and terminal latents through the predicted velocity.
  • Scope: The same methods can be applied to the OPSD of flow-matching models to improve model performance.

C.5. Modality Mimicry with On-policy Distillation

The authors compare fixed-teacher and self-distillation variants for modality mimicry, finding the paper’s original mutual-distribution training paradigm more suitable for this setting.

  • A fixed-weight pretrained T2IV teacher was added to compare original-capability and post-training references for T2I modality mimicry.The modality mimic generation loss used MSE between teacher-derived T2V first-frame latents and T2I latents.
  • The fixed-teacher approach showed gaps in convergence and performance compared with the paper’s original self-distillation paradigm.A gradient-stopped self-distillation validation likewise showed significant degradation in convergence speed and modality-mimicry performance.
  • The experiments demonstrate on-policy distillation feasibility for flow-matching models while favoring mutual generation-distribution mimicry for this scenario.The authors note that related experiments could be adapted to other on-policy distillation settings, such as using an I2T teacher for T2I prompt faithfulness.

C.6. Perceptual Guidance from Clean Images

The authors explored clean-image perceptual guidance to improve stylistic responsiveness across image and video modalities, but did not include this incremental method in the paper’s final model.

  • A style perceptual feature loss compares clean-image features with decoded T2I and sampled T2V estimated-original latents.The method uses cosine similarity features from the CSD model and samples one of the first four T2V frames.
  • The clean-image and latent estimates were evaluated only in the non-high-noise timestep interval [0, 900).This restriction was intended to preserve sufficient clarity of estimated original images in image space.
  • The loss accelerated convergence for stylized T2V and improved style accuracy and aesthetics for stylized T2I generation.The authors also observed improved aesthetics and prompt faithfulness under non-stylized prompts.
  • Figure 33 reports 61-frame, 832 × 480 R2V examples generated without R2V or global R2I training samples.The examples show prompt-following video generation conditioned on editing prompts and reference images.

C.7. Additional Reference Injection

The authors tested cross-attention reference injection, which improved reference preservation to some extent but was excluded to keep the architecture clean.

  • The additional cross-attention method injected per-reference visual representations from DINOv2-giant and OpenCLIP encoders.The representations were extracted from each reference element’s first frame.
  • The method was omitted because its architectural additions conflicted with the design principle of keeping the model architecturally clean.The authors nevertheless observed some improvement in reference preservation.

C.8. Reference-to-Video task

The authors explored Reference-to-Video by constructing reference-conditioned video samples and found R2V ability even without direct R2V or global R2I training data.

  • Global R2I samples were converted into motion videos with static or dynamic camera movements, with motion captions added to form R2V targets.A separate S2V construction used detected, cropped, rotated, or flipped first-frame subjects as references for T2V-derived samples.
  • The model exhibited a degree of R2V capability whether R2V and global R2I tasks were excluded or included without T2V-derived S2V samples.This observation motivated treating the R2V setting as combining temporal editing and temporal generation abilities.
  • Without R2V training samples, the model still preserved reference information across generated frames while producing natural, prompt-following actions.The authors attribute this observation to transfer from T2V and I2V training together with the pixel-pair temporal warped flow field.
  • R2V was excluded from the main training verification to avoid confounding the test of learning video editing from image editing samples alone.The concern arose because some R2V samples were partially constructed from T2V samples.

C.9. Addressing Excessive Mimicry Parameters

The authors identify excessive mimicry as a failure mode in which peaked, similarly positioned logits make the modality mimicry loss artificially small. They introduce an additive-dimension strategy that removes this phenomenon while preserving mimicry effects without adverse inference side effects.

  • A shared unusually large logit at one position can make the T2V and T2I distributions similarly peaked, artificially reducing the modality mimic generation loss.The authors describe this as an undesirable natural learning procedure.
  • Temperature scaling, logit normalization, regularization, and entropy maximization did not produce a discernible effect in experiments.Large temperatures blurred inference generations, whereas smaller temperatures did not noticeably mitigate the phenomenon.
  • The strategy improves modality mimicry, eliminates the sharp-logit phenomenon, preserves the original effect, and introduces no adverse inference side effects.The authors describe the extreme settings producing this issue as having limited practical attention.
  • The additive-dimension strategy appends an extra dimension to both modality logits, using c_add set to k times the observed maximum value.The authors set k = 1.3 when continuing from an affected model and k = 1.1 when post-training directly from Wan2.1-T2V-1.3B.
  • The added constant gives softmax an extremely high-probability logit position, allowing the remaining logits to learn gently without undue pressure.The sharp-value phenomenon gradually disappears during continued training or does not emerge during post-training from the pretrained model.
Loading 2607.18227v1…