Source-linked AI summary
MotionEdit: Benchmarking and Learning Motion-Centric Image Editing
Yixin Wan, Lei Ke, Wenhao Yu, Kai-Wei Chang, Dong Yu
TL;DR
Motion-centric image editing remains poorly served by appearance-focused datasets and models that struggle to modify actions and interactions while preserving visual consistency. MotionEdit provides video-derived data and a benchmark, and MotionNFT uses optical-flow alignment rewards in post-training. Experiments report consistent gains across generative quality, motion alignment, and preference win rate on two base models.
Problem
Method
MotionNFT post-trains editing models with optical-flow rewards that compare model-edited motion against ground-truth motion.
Source: §1 · p. 3, §7.3.1 · p. 12Results
MotionNFT delivers consistent gains across generative quality, motion alignment, and preference win rate on FLUX.1 Kontext and Qwen-Image-Edit.
Source: §6 · p. 9Takeaways & Limitations
The results show that motion-aware post-training can improve motion editing across two strong base models.
Source: §6 · p. 9Multi-subject interactions and preserving one subject’s identity while changing its motion remain challenging cases.
Source: §9.2.3 · p. 16
Abstract
from arXiv · showhide
We introduce MotionEdit, a novel dataset for motion-centric image editing-the task of modifying subject actions and interactions while preserving identity, structure, and physical plausibility. Unlike existing image editing datasets that focus on static appearance changes or contain only sparse, low-quality motion edits, MotionEdit provides high-fidelity image pairs depicting realistic motion transformations extracted and verified from continuous videos. This new task is not only scientifically challenging but also practically significant, powering downstream applications such as frame-controlled video synthesis and animation. To evaluate model performance on the novel task, we introduce MotionEdit-Bench, a benchmark that challenges models on motion-centric edits and measures model performance with generative, discriminative, and preference-based metrics. Benchmark results reveal that motion editing remains highly challenging for existing state-of-the-art diffusion-based editing models. To address this gap, we propose MotionNFT (Motion-guided Negative-aware Fine Tuning), a post-training framework that computes motion alignment rewards based on how well the motion flow between input and model-edited images matches the ground-truth motion, guiding models toward accurate motion transformations. Extensive experiments on FLUX.1 Kontext and Qwen-Image-Edit show that MotionNFT consistently improves editing quality and motion fidelity of both base models on the motion editing task without sacrificing general editing ability, demonstrating its effectiveness. Our code is at https://github.com/elainew728/motion-edit/.
1. Introduction
MotionEdit formulates motion-centric editing as a distinct task and introduces a video-derived dataset and benchmark for evaluating it. MotionNFT adds motion-aware rewards to post-training, while benchmark results show that motion editing remains challenging for existing models and improves with the proposed framework.
1. Introduction
- Motion editing changes subjects’ actions, poses, or interactions while preserving visual consistency, addressing gaps in datasets focused on appearance or containing sparse, low-quality motion edits.The task is relevant to frame-controlled video generation and character animation.Source: §1 · p. 2
- MOTIONEDIT pairs video-derived input and target images to capture realistic action and interaction changes while preserving identity, background, and style.Its examples span motion subcategories including posture, orientation, and interaction changes.Source: §1 · p. 2
- MotionEdit-Bench evaluates edits with optical-flow-based discriminative measures, MLLM-based generative metrics, and pairwise preference win rates.Benchmark results indicate that motion editing remains challenging for most open-source image-editing models.Source: §1 · pp. 2–3
- MotionNFT yields over 10% improvement in overall quality and over 12% on pairwise win rates when applied to FLUX.1 Kontext.The framework uses motion alignment between input-edit and input-ground-truth optical flows to guide training.Source: §1 · p. 3
2. Related Works
Prior image-editing work handles static appearance changes more readily than motion edits, while existing datasets provide limited coverage or low-quality motion targets. Existing reinforcement-learning post-training is also described as motion-agnostic.
Image Editing
- Existing datasets mostly target static transformations, while motion-edit datasets are small and often provide unfaithful or physically implausible targets.Such limitations can undermine training supervision and evaluation reliability.Source: §2 · p. 3
Motion Estimation in Images
Reinforcement Learning for Image Generation
- Existing RL-based post-training frameworks emphasize semantic correctness and visual details but do not supervise how subjects and objects should move.DiffusionNFT [41] contrasts positive and negative generations; UniWorld-V2 [15] adds MLLM-based scoring for aspects such as prompt compliance and style fidelity.Source: §2 · p. 4
3. Dataset Construction
The dataset construction process defines motion editing, mines and validates video frame pairs, rewrites instructions, and assembles a diverse benchmark. MOTIONEDIT contains substantially larger motion changes than the comparison datasets described.
3.1. Problem Definition and Categorization
- Motion image editing changes instructed subject or object actions, poses, or interactions while preserving appearance, background, and viewpoint consistency.The task also requires physically plausible poses or interactions and fine-grained spatial and kinematic understanding.Source: §3.1 · p. 4
3.2. Dataset Construction Pipeline
- The pipeline mines candidate video frames, samples the first and last frames of three-second segments, and filters pairs for scene stability, meaningful motion, subject consistency, and visual quality.Candidate videos come from ShareVeo3 [31] and the KlingAI Video Dataset, selected after conventional action datasets were judged unsuitable for faithful pairs.Source: §3.2 · p. 4
3.3. Editing Prompt Construction
- Validated motion summaries are rewritten as clear, imperative user-style editing instructions aligned with the observed transition.Figure 4 depicts frame extraction, MLLM filtering, and instruction rewriting.Source: §3.3 · p. 4, Figure 4 · p. 5
3.4. Dataset Statistics
- MOTIONEDIT contains 10,157 frame pairs, with 9,142 training examples and 1,015 evaluation examples in MOTIONEDIT-BENCH.The dataset draws 6,006 samples from Veo-3 and 4,151 from KlingAI and includes six motion-edit types, including pose, locomotion, object state, orientation, and interactions.Source: §3.4 · p. 5
3.5. Data Motion Magnitude Comparison
- Figure 6 reports average input-target motion magnitude across six editing datasets: prior datasets are typically around 0.05, while MOTIONEDIT is 0.19—5.8× MagicBrush and OmniEdit, and 3× UltraEdit.Source: §3.5 · p. 5
4. Learning Motion Image Editing
MotionNFT extends negative-aware fine-tuning with a motion reward derived from the alignment of predicted and ground-truth optical flow. The reward combines motion magnitude, direction, and movement regularization to guide model updates.
4.1. Preliminaries
- DiffusionNFT trains toward a positive velocity and away from a negative velocity, with both formed as linear combinations of old and current policies.Its calibrated optimality reward normalizes raw rewards to stabilize learning and maintain positive/negative assignments across prompts and reward models.Source: §4.1 · p. 6
4.2. MotionNFT: Motion-Aware Reward for NFT
- MotionNFT scores alignment by comparing optical flow from the input to the model edit with flow from the input to the ground-truth edit.Both flow fields are normalized by the image diagonal for scale consistency across resolutions.Source: §4.2 · p. 6
- The motion reward includes magnitude deviation, direction misalignment weighted by ground-truth motion, and a movement penalty for edits with too little motion.The magnitude term uses a robust ℓ1 distance with q in (0, 1) to suppress outliers.Source: §4.2 · p. 6
- MotionNFT combines the three motion terms into a normalized, clipped reward, converts it to six discrete levels, and uses it to update the policy under the DiffusionNFT objective.The composite distance weights magnitude, direction, and movement terms with constants α, β, and λ_move.Source: §4.2 · p. 6
5. Experiments
The experiments evaluate nine open-source editors on MotionEdit-Bench and test MotionNFT’s effects on motion editing and general editing ability. Results span generative ratings, motion alignment, pairwise preference, qualitative examples, and comparisons with other training approaches.
5.3. Quantitative Evaluation Results
- FLUX.1 Kontext gains Overall 3.84→4.25, MAS 53.73→55.45, and win rate 57.97%→65.16%; Qwen-Image-Edit improves on all three.For Qwen-Image-Edit, Overall rises from 4.65 to 4.72, MAS from 56.46 to 57.23, and win rate from 72.99% to 73.87%.Source: Table 1 · p. 7, §5.3 · p. 8
- Among existing methods, Step1X-Edit and BAGEL perform strongest on MotionEdit-Bench, while AnyEdit and MagicBrush perform poorly across generative and discriminative metrics.MotionNFT-trained FLUX.1 Kontext and Qwen-Image-Edit achieve the best overall results.Source: Table 1 · p. 7
5.1. Experimental Setup
- Nine open-source editors are benchmarked; MotionNFT training uses FLUX.1 Kontext and Qwen-Image-Edit with equally weighted optical-flow and MLLM rewards.The optical-flow reward uses UniMatch for motion-level guidance.Source: §5.1 · pp. 6–7
5.2. Evaluation Metrics
- Evaluation pairs MLLM Fidelity, Preservation, Coherence, and Overall ratings with optical-flow MAS for magnitude and direction consistency, plus pairwise win rate.MAS assigns zero when predicted motion is nearly static relative to ground truth.Source: §5.2 · p. 7, §5.3 · p. 7
5.4. Qualitative Evaluation Results
- Qualitative examples show baseline Qwen-Image-Edit often misses instructed actions, while MotionNFT produces closer target motion while retaining identity and scene context.Examples include rotating character directions and adjusting limb and torso positions for bending or turning actions.Source: Figure 8 · p. 8, §5.4 · p. 8
5.5. Ablation Studies
- On ImgEdit-Bench’s eight subtasks, MotionNFT maintains or improves general-editing performance for both base models, with higher overall scores.The authors report no trade-off in general editing quality.Source: §5.5 · p. 9, Table 2 · p. 9
- Against MLLM-only RL, MotionNFT achieves higher overall edit quality, motion alignment, and win rates across both base models.The comparison supports the value of adding optical-flow-based motion guidance to the reward.Source: §5.5 · p. 9, Table 3 · p. 9
6. Conclusion
The conclusion presents MotionEdit as a dataset and benchmark for motion image editing, then proposes MotionNFT to improve that task. Experiments report consistent gains across generative quality, motion alignment, and preference win rate on two base models.
6. Conclusion
- MotionNFT integrates an optical-flow motion reward that supervises motion magnitude and direction; experiments show consistent gains across quality, alignment, and win rate on FLUX.1 Kontext and Qwen-Image-Edit.The conclusion frames the task as modifying subject actions and interactions while preserving identity and scene consistency.Source: §6 · p. 9
Supplementary Material
The supplementary material adds method and experiment details, metric design and validation, ablations, and qualitative comparisons with open-source and commercial models.
Supplementary Material
- Supplementary material covers method design, experimental setup, metric design and human validation, ablations, and qualitative model comparisons including failure cases.It also points toward future research directions.Source: Supplementary Material · p. 12
7. Additional Method Details
The additional method details explain flow-matching and negative-aware fine-tuning, then specify MotionNFT’s optical-flow reward for aligning generated motion with ground truth.
7.1. Preliminaries
- Flow matching predicts a velocity field from noise toward clean samples and uses a deterministic ODE for inference.The noisy latent interpolates between clean data and Gaussian noise; DiffusionNFT further models positive and negative velocities and transforms raw rewards into optimality rewards.Source: §7.1 · p. 12
7.2. Diffusion Negative-aware Finetuning (NFT)
- DiffusionNFT combines old and current policies into implicit positive and negative velocities, then converts raw rewards into optimality rewards.The positive policy uses weights (1 − β) and β, while the negative policy uses (1 + β) and −β.Source: §7.2 · p. 12
7.3.1. Motion Calculation
- MotionNFT compares optical flow from the input to the model edit against flow from the input to ground truth, normalizing displacements by image diagonal.The reward combines motion magnitude consistency, motion direction consistency, and movement regularization.Source: §7.3 · p. 12, §7.3.1 · pp. 12–13
7.3.2. Reward Calculation
- Magnitude consistency uses a robust ℓ1 distance, while directional consistency measures flow alignment in regions with non-trivial motion.The robust distance suppresses large flow outliers while preserving sensitivity to meaningful deviations; directional error is cosine-based and weighted by ground-truth motion magnitude.Source: §7.3.2 · p. 13
- Movement regularization penalizes nearly static edits, and the combined magnitude, direction, and movement terms are normalized into a scalar training reward.The reward is quantized to six equally spaced levels to approximate discrete human ratings, then group-normalized to update the policy.Source: §7.3.2 · p. 13
8. Additional Evaluation Experiment Details
These details specify the reward metric, model training and inference settings, and a human comparison study. Human judgments show substantial agreement with the generative metric.
8.1.1. Reward and Metric
- The reward combines motion magnitude, direction, and movement regularization, with weights 0.7, 0.2, and 0.1; near-static predictions receive the minimum MAS score.For MAS, the near-static failure threshold is set to ρmin = 0.01.Source: §8.1.1 · p. 13
8.1.2. Model Training
- Training uses learning rate 3e −4 and batch size 2, with 300 steps for FLUX.1 Kontext and 210 for Qwen-Image-Edit.NFT sampling uses 6 inference steps, 8 images per prompt, and 24 groups; the KL loss weight is 0.0001 and guidance strength is 1.0.Source: §8.1.2 · p. 14
8.1.3. Model Inference
- Qwen-Image-Edit and trained checkpoints use 28 inference steps, while other open-source models use model-specific or default repository settings.UniWorld uses 25 steps, AnyEdit 100, UltraEdit 50, and Step-1X 28.Source: §8.1.3 · p. 14
8.2. Human Evaluation for Generative Metrics
- The human evaluation used voluntary U.S.-based college or graduate students proficient in English and familiar with AI research.Annotators were told their anonymized results could be used for research purposes.Source: §8.2 · p. 14
8.2.1. Annotation Interface and Instructions
- Annotators compared two model outputs against an input, instruction, and ground-truth reference, judging motion correctness, identity preservation, and visual coherence.The study sampled 100 benchmark entries, and each model pair was rated by three annotators; ground truth was a reference, not a pixelwise target.Source: §8.2.1 · p. 14
8.2.2. Human Evaluation Results.
- Fleiss’ κ was 0.607 among human annotators and 0.574 between annotators and the generative metric, indicating substantial agreement.The metric averaged Fidelity, Preservation, and Coherence.Source: §8.2.2 · p. 14
9. Additional Evaluation Results
A balanced semantic and optical-flow reward performs best in the reported ablation, while comparisons show stronger motion grounding alongside persistent challenges in complex scenes.
9.1. Ablation Studies
- The 0.5:0.5 reward mix achieves the highest performance across metrics, scoring 4.25 Overall on FLUX.1 Kontext and 4.72 Overall on Qwen-Image-Edit.Motion-only training degrades performance, while MLLM-only training is consistently outperformed by the combined approach.Source: Table 4 · p. 15, §9.1 · p. 15
- MotionNFT improves final MAS across both backbones, whereas MLLM-only reward training can regress during training, including a mid-training drop for Qwen-Image-Edit.MAS measures magnitude and directional alignment between model-edited motion and the ground-truth target.Source: §9.1 · p. 15, Figure 9 · p. 15, Figure 10 · p. 15
9.2.3. Failure Analysis and Limitations
- Multi-subject edits and preserving uninvolved subjects remain difficult for MotionNFT and commercial models in complex scenes.Examples include positioning an orca relative to a polar bear and preserving additional subjects in a tent scene.Source: §9.2.3 · p. 16
9.3. Speed and Inference Cost
- MotionNFT adds no inference-time cost; single-entry inference takes about 48 seconds with FLUX.1 and 98 seconds with Qwen-Image-Edit on one NVIDIA GPU.The reported timings use 28 sampling steps.Source: §9.3 · p. 16
9.1. Ablation Studies
- In qualitative comparisons, MotionNFT executes motion changes more precisely than base and MLLM-only models while preserving appearance and scene setting.The cited cases include misplaced hands, incorrect motion direction, and failures to preserve a subject’s appearance.Source: Figure 11 · p. 17, Figure 12 · p. 18
9.2.1. Comparison with Open-Source Models
- Against open-source baselines, MotionNFT more accurately grounds challenging motion edits, while baselines show editing inertia and motion misalignment.The comparison highlights cases where baselines fail to change a pose or move the wrong arm.Source: Figure 13 · p. 19
9.2.2. Comparison with Closed-source Models
- Commercial baselines often struggle with complex motion changes or visual consistency, while MotionNFT follows the instructions with geometric alignment to ground truth.The cited cases include failure to displace a car and generation of an artifact foot.Source: Figure 14 · p. 20