PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionCoach: Constructing Multimodal Guidance for Video Professional Motion Editing

Siyi Qian, Hanqing Yang, Weihuang Lin, Zihao Yang, Yichen Lu, Yuanhong Zheng, Sheng Li

arXiv:2610.12963v1cs.CVcs.AIcs.MM

TL;DR

Professional motion editing demands technique and temporal structure that short motion names do not reliably convey. MotionCoach constructs multimodal guidance from verified reference clips and grounded prompts. On VPMEBench, it improves overall scores for Seedance 2.0 and MiniMax-H3 without retraining the editing models.

  • Problem

    Professional motion editing requires fine-grained technique and temporal structure, but existing editors can generate incorrect motions from named instructions, while reliable guidance acquisition and verification remain unresolved.

    Source: §1 · p. 1

  • Method

    MotionCoach is a trained multimodal agent that uses tools and self-verification to construct grounded prompts and reference clips, trained with SFT and GRPO-based agentic reinforcement learning.

    Source: §1 · pp. 1–2

  • Results

    55.1 to 65.6 for Seedance 2.0 and 51.2 to 64.5 for MiniMax-H3: MotionCoach improves overall VPMEBench scores.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MotionCoach improves professional motion editing on frozen video editing models by supplying reliable and sufficient multimodal guidance.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video editing models achieve strong performance in modifying appearance, backgrounds, styles, and motions, empowered by large-scale training data. Beyond general scenarios, professional motions are significant in performance, sports, and martial arts. This paper focuses on professional motion editing, which involves fine-grained motion technique and temporal structure. Given professional motion names but lacking detailed descriptions, existing models often fail to generate professional motions, such as incorrectly lifting the front wheel under a Stoppie instruction. To address this challenge, we introduce MotionCoach, a trained multimodal agent that uses multi-turn tools and performs self-verification, producing a grounded prompt and a reference clip as reliable and sufficient multimodal guidance to inject external professional motion knowledge into video editing models. To supply training data, we build a data pipeline to construct trajectories. We cold-start the agent on MotionCoach-SFT-10k and apply Group Relative Policy Optimization (GRPO) on MotionCoach-RL-2k with rewards on the motion guidance prompt, the motion outcome video, and the format. For evaluation, we introduce VPMEBench, comprising 314 professional motion editing tasks with fine-grained references and annotations. Extensive experiments show that MotionCoach improves the overall score from 55.1 to 65.6 on Seedance 2.0 and from 51.2 to 64.5 on MiniMax-H3. Learning to construct multimodal guidance enables precise professional motion editing without retraining video editing models.

1 Introduction

Professional motion editing requires technically correct motions with appropriate temporal structure, but short motion names and instructions often leave those details unspecified. MotionCoach constructs verified reference clips and grounded prompts, and experiments report improved VPMEBench scores for both Seedance 2.0 and MiniMax-H3.

1 Introduction

  • Professional motions require fine-grained technique and temporal structure, yet existing editors can produce incorrect motions from named instructions.A Stoppie request can yield a Wheelie, lifting the wrong wheel.Source: §1 · p. 1, Figure 1 · p. 2
  • MotionCoach searches for demonstrations, inspects and clips a target-motion interval, then self-verifies correctness, visibility, and completeness before producing a grounded prompt and reference clip.Verification can trigger reselection or temporal adjustment.Source: §1 · pp. 1–2
  • A teacher-generated trajectory pipeline with VLM checks and human spot-checking supplies MotionCoach-SFT-10k and MotionCoach-RL-2k for two-stage training.SFT establishes formatting and tool use; GRPO-based RL uses prompt, outcome-video, and format rewards.Source: §1 · p. 2, Figure 2 · p. 3
  • VPMEBench provides 314 human-verified editing tasks across 25 dance, sports, and martial-arts categories, with references and annotations for evaluation.Its references include verified clips and semantic keyframes; annotations cover temporal structure, technique, and negative constraints.Source: §1 · p. 2
  • 55.1 to 65.6 for Seedance 2.0 and 51.2 to 64.5 for MiniMax-H3: MotionCoach improves overall VPMEBench scores.Scores are on a 100-point scale.Source: §1 · p. 2
  • MotionCoach combines a grounded prompt and reference clip as reliable, sufficient guidance for editing models.The paper identifies the construction of reliable and sufficient guidance as an unresolved challenge.Source: §1 · pp. 1–2

2 Related Work

Prior video-editing work targets appearance, scene, and style changes, while professional motion editing remains difficult. Agentic methods use tools, external knowledge, or learned policies, but motion technique and temporal structure remain unverified.

Video editing

  • Instruction-guided editing, scene and clothing changes, and reference-conditioned editing are among existing video-editing approaches.Examples include InsV2V, Ditto, Lucy-Edit-Dev, VACE, and Kiwi-Edit.Source: §2 · p. 3

Agentic visual generation and editing

  • Agentic visual methods orchestrate pretrained models with planning, feedback, and external knowledge, while another line trains policies linking decisions to visual outcomes.Examples include RS-Gen for image-generation retrieval and VideoAgent for video-editing planning, search, and trimming.Source: §2 · p. 3
  • Video-generation and editing agent methods remain comparatively underexplored, and motion technique and temporal structure stay unverified.These limitations motivate learning to construct editing conditions.Source: §2 · p. 3

3 Method

The method builds task-specific trajectories and a fine-grained benchmark, then trains MotionCoach to retrieve, verify, and package professional-motion references as prompt-and-clip guidance for a frozen editor.

3.1 Data Construction

  • The data pipeline prepares source/edit pairs, constructs and filters tool-use trajectories, and builds VPMEBench with 314 tasks across 3 top-level, 10 second-level, and 25 leaf categories.Each task includes visual references and textual annotations for motion professionalism (MP), physical plausibility (PP), and video quality (VQ).Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Agentic Framework

  • MotionCoach uses search, inspection, and clipping to select professional-motion references, verify a clear interval, and provide a grounded prompt and clip to a frozen video editor.If a clip fails self-verification, the agent re-clips the video or selects another candidate.Source: §3.2 · p. 5
  • The grounded prompt describes motion technique, temporal structure, and errors to avoid, preserves other source content, and uses a dense-temporal, sparse-stage, or key-pose format.Source: §3.2 · pp. 5–6

3.3 Training Strategy

  • Training cold-starts on MotionCoach-SFT-10k for basic tool use and formatting, then applies GRPO on MotionCoach-RL-2k to improve guidance and editing performance.Source: §3.3 · p. 6
  • The RL rewards assess prompt correctness and completeness, the edited video against a teacher reference, and valid tool calls and structured answers.The prompt reward scores correctness, specificity, structural completeness, edit scope, and clarity; the video reward scores motion professionalism, physical plausibility, visual quality, and edit preservation.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate MotionCoach on VPMEBench with frozen video editors and report gains in professional motion editing. Ablations examine the effects of training, guidance inputs, and agent components.

4.1 Experimental Setup

  • Evaluation covers 314 VPMEBench samples and frozen MiniMax-H3 and Seedance 2.0; human preference agreement reaches 0.846 QWK overall and exceeds 0.73 on every dimension.Training uses SFT followed by GRPO; rollout video generation uses MiniMax-H3.Source: §4.1 · pp. 6–7, Table 9 · p. 9

4.2 Main Results on VPMEBench

  • Seedance 2.0 and HappyHorse 1.0 reach only 55.1 and 60.1 overall, while MotionCoach improves frozen MiniMax-H3 across reported metrics and transfers across most Seedance dimensions.The table evaluates MotionCoach on frozen MiniMax-H3 and Seedance 2.0 against existing editors using short edit prompts.Source: §4.2 · p. 7, Table 1 · p. 7
  • Across dance, martial arts, and sports, MotionCoach produces accurate body configurations and temporal structures where existing models retain source motion or misread motion names.Figure 5 presents qualitative results.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 Experimental Analysis

  • Tool access alone gives inconsistent gains, while SFT improves both base editors and RL further refines the SFT policy.The comparison attributes overall gains to learning to construct useful guidance rather than merely executing the workflow.Source: Table 2 · p. 7, §4.3 · p. 8
  • Removing the grounded prompt costs 13.2/15.9 points on Seedance/MiniMax-H3, versus 5.8/0.8 points when removing the reference clip.The prompt carries motion technique and temporal structure; the clip supplies a complementary visual reference.Source: Table 3 · p. 7, §4.3 · p. 8
  • Removing video, focused clipping, or verification lowers Overall by 10.7/13.2, 8.2/10.4, or 7.5/9.9 points, respectively.The results indicate complementary contributions from temporal video reference, focused clip selection, and explicit self-verification.Source: Table 4 · p. 7, §4.3 · pp. 8–9
  • Clip execution rises from 62.2% to 98.3%, and training raises trajectory success to 99.7%.The evaluation tracks query accuracy, tool-call success, Pass@1, and trajectory success.Source: Table 5 · p. 7, §4.3 · p. 9
  • The 0.7:0.3 prompt-to-video reward ratio achieves 65.61 Overall, while task-specific training beats backbone scaling and transfers to unseen settings.The trained 8B policy exceeds the untrained 27B policy by 8.16 points on Seedance 2.0 and 8.39 points on MiniMax-H3; on unseen IVEBench subject-motion editing, Total rises from 48.0 to 53.8.Source: §4.3 · p. 9, Table 7 · p. 9, Table 8 · p. 9

5 Conclusion

The paper addresses professional motion editing, where edited videos must capture motion technique and temporal structure. MotionCoach constructs multimodal guidance for frozen editors, and experiments report improved professional motion editing without retraining those editors.

5 Conclusion

  • MotionCoach uses multi-turn tools and self-verification to provide frozen video editors with a reference clip and grounded prompt as multimodal guidance.The paper also builds a trajectory data pipeline and VPMEBench, and trains the policy with agentic reinforcement learning.Source: §5 · p. 9

A.1 Benchmark

VPMEBench supplies professional-motion editing tasks with human-verified references and fine-grained annotations. Its benchmark also includes a human validation study comparing outputs from frozen editors with and without MotionCoach guidance.

A.1 Benchmark

  • VPMEBench contains 314 tasks across 3 top-level, 10 second-level, and 25 leaf categories spanning dance, sports and fitness, and martial arts.Dance accounts for 144 tasks, sports and fitness 110, and martial arts 60.Source: §A.1 · p. 12
  • Each task includes a human-verified five-second reference clip, temporal-structure annotations, one to six semantic keyframes, two to five technique criteria, and two to four negative constraints.Temporal annotations use dense, sparse, or key-state forms, represented by 191, 76, and 47 tasks, respectively.Source: §A.1 · p. 12
  • In a stratified sample of 100 paired outputs, three experienced annotators judged each pair, and automatic-human agreement reached QWK 0.803, 0.733, 0.943, and 0.846 for MP, PP, VQ, and Overall.The sample comprised 50 pairs from each editor, and confidence intervals used 10,000 task-clustered bootstrap resamples.Source: §A.1 · p. 12

A.2 Additional Results

Additional results show MotionCoach handling dance, sports, and martial-arts edits while preserving source content, with failures on challenging terrain and subtle articulation.

A.2 Additional Results

  • The first eight Figure 7 cases show accurate target-motion execution across dance, sports, and martial arts while preserving other content.The last two cases show limitations in challenging settings.Source: §A.2 · p. 12, Figure 7 · p. 14
  • The dance examples preserve characteristic limb configurations, shoulder movement, torso inclination, and leg trajectories through coordinated motion and smooth pose transitions.The Taekwondo case includes two kicks with an intermediate rechambering phase, while Kickflip covers crouching, takeoff, airborne rotation, and landing.Source: §A.2 · p. 12
  • Complex terrain can leave a G-turn incomplete with insufficient turning radius, while subtle hip rotation remains difficult in the U-shaped Hip Swing.Figure 6 presents representative VPMEBench tasks.Source: §A.2 · p. 12, Figure 6 · p. 13

A.3 Agent Prompts and Tool Schemas

This section specifies MotionCoach’s tool-mediated workflow for finding, inspecting, clipping, and self-checking professional-motion references before generating a grounded editing prompt, along with the final response format. It also describes separate judges and rubrics for prompt quality, edited-video outcomes, and VPMEBench evaluation.

A.3 Agent Prompts and Tool Schemas

  • The appendix provides the agent and judge prompts, the motion-guidance prompt reward judge, tool schemas, per-round output format, final-answer schema, and motion-outcome video reward judge.Source: §A.3 · p. 15

A.3.1 Agent System Prompt

  • The agent normalizes the target motion name, searches three candidates, prioritizes exact matches, inspects the full reference, and clips its clearest useful segment to at most five seconds.Candidate selection may use motion semantics and bilingual names when no exact match exists; the inspected video provides visual and, when available, audio evidence.Source: §A.3.1 · p. 15
  • After clipping, the agent must assess motion visibility, mechanics, completeness, clarity, and relevance, then reclip or inspect another candidate if needed.A clip must pass this self-check before the agent can provide its final answer.Source: §A.3.1 · p. 15
  • The final prompt combines the full reference’s explanatory evidence, the checked clip’s visible motion, and Video 1’s properties to preserve, without inventing unsupported motion phases.The source video supplies only properties to retain; the reference informs the target motion, while the clip conveys its order, key states, and mechanics.Source: §A.3.1 · pp. 15–16
  • Regular rounds allow at most one tool call, and the final answer must contain exactly a grounded_prompt and one reference_videos entry identifying the most recent clip that passed self-check.The final prompt follows mode-specific timestamp rules and includes the exact motion, mechanics, constraints, and visible source-preservation properties.Source: §A.3.1 · pp. 16, 18
  • The agent selects dense, sparse-stage, or single-state description according to the clip, then specifies motion mechanics and errors while preserving visible source-video properties.Dense mode uses chronological intervals; sparse mode follows actual phase boundaries; single-state mode describes a pose without timestamps or invented progression.Source: §A.3.1 · pp. 16–17

A.3.2 Evaluation Judge Prompt

  • The VPMEBench judge evaluates motion professionalism, physical plausibility, video quality, and edit preservation on a 0, 0.5, or 1 scale, accepting only schema-valid JSON judgments.Each task includes the source video, edit instruction, motion annotations, verified reference clip, semantic keyframes, and generated video; the prompt configures Gemini 3.5 Flash as judge.Source: §A.3.2 · p. 18
  • The evaluation judge uses reference video for temporal mechanics and reference images for decisive poses, while accepting professionally correct equivalent executions rather than pixel-level imitation.For static targets, it judges pose and stability without inventing temporal stages; dimensions are scored independently as 0, 0.5, or 1.Source: §A.3.2 · p. 18
  • Motion professionalism is judged by whether the subject performs the requested action and matches reference mechanics; physical plausibility covers continuity, balance, anatomy, contacts, and natural transitions.The rubric penalizes temporal or anatomical breakdowns such as stutter, teleportation, sliding, and missing or duplicated limbs.Source: §A.3.2 · pp. 18–19
  • Video quality assesses visual clarity and degradation, while edit preservation checks whether identity and scene remain intact apart from necessary motion-related changes.The rubric penalizes blur, flicker, artifacts, unnecessary replacement, drift, deletion, added content, and camera or scene redesign.Source: §A.3.2 · p. 19

A.3.3 Motion Guidance Prompt Reward Prompt

  • The prompt-reward judge scores six weighted dimensions, normalizes their sum to [0, 1], caps hard-error cases at 0.2, and uses deterministic decoding with valid-JSON acceptance.The dimensions cover instruction fidelity, source-video grounding, motion correctness and specificity, template conformance, preservation and edit scope, and clarity and non-hallucination.Source: §A.3.3 · p. 19
  • The prompt-reward judge evaluates semantic correctness against the source, instruction, and teacher prompt, which guides structure and coverage rather than exact wording.Unsupported source content is disallowed, and copying the teacher’s structure does not compensate for content that contradicts the source or instruction.Source: §A.3.3 · p. 20
  • The prompt judge evaluates instruction fidelity, source-video grounding, preservation and edit scope, and clarity/non-hallucination; hard errors include changing the wrong subject or action, missing requested motion, contradicting instructions or visible video, or producing an unusable editing prompt.Source: §A.3.3 · p. 20
  • Template conformance requires source-preservation constraints and a motion description whose form fits the target: ordered stages for temporal motion or a single stable key-state description for static motion.The rubric accepts either form and scores its fit, not the presence of timestamps; motion descriptions should specify executable mechanics such as body parts, phases, direction, speed, rhythm, and interactions.Source: §A.3.3 · p. 20

A.3.4 Motion Outcome Video Reward Judge

  • The video-reward judge evaluates frozen-editor outputs on four dimensions scored 0, 0.5, or 1, using source, teacher reference, generated video, and edit instruction.It decodes at temperature 0 and requires a single-line JSON judgement.Source: §A.3.4 · p. 21

A.4 Multi-Turn Agent Loop Example

Figure 8 illustrates a multi-turn agent loop in which inspection evidence guides a corrected clip selection.

A.4 Multi-Turn Agent Loop Example

  • Figure 8 illustrates a MotionCoach trajectory for the multi-turn loop described in Section 3.2.The agent uses search, inspect, and clip tools.Source: §A.4 · p. 21
  • The agent recognizes that its first clip shows the wrong motion and clips again once.The reclip uses on-screen text evidence already returned by inspect.Source: §A.4 · p. 21

A.5 Training and Reward Details

The training-details passage identifies the policy and states that it is trained in two stages.

A.5 Training and Reward Details

A.5.1 Supervised Fine-Tuning

The appendix describes the supervised fine-tuning setup and the subsequent agentic reinforcement-learning configuration, including the agent loop and training infrastructure.

A.5.1 Supervised Fine-Tuning

  • SFT uses full-parameter optimization with LLaMA-Factory and DeepSpeed ZeRO-3; Table 11 reports its principal hyperparameters.Source: §A.5.1 · p. 21
  • The multi-turn agent loop alternates thinking, tool calls, and tool responses across search, inspect, and clip, then returns a grounded prompt and verified reference clip.Source: Figure 8 · p. 22

A.5.2 Agentic Reinforcement Learning

  • RL starts from the final SFT checkpoint and applies full-parameter GRPO with verl, using separate nodes for policy training, video-editor rollouts, and Qwen3-VL-30B-A3B reward judging.The policy-training and rollout workers occupy two eight-GPU nodes; a third eight-GPU node hosts the frozen video editor.Source: §A.5.2 · p. 22

Related papers

Loading 2610.12963v1…