What Moves? Localized Motion Representations for Compositional Scene Control

• Global motion embeddings entangle entities, while pixel-level flow or trajectories lack semantic structure, limiting object-level manipulation.Source: §1 · p. 2
• Encoding the full video while conditioning on region masks preserves global context and localizes motion, unlike cropping or post-hoc feature masking.Source: §1 · p. 2
• Across motion transfer and localized action classification, the approach improves controllability and motion understanding over global representations localized by cropping or feature masking.Source: §1 · p. 2
• The paper introduces persistent, region-specific motion embeddings conditioned on spatial regions.Source: §1 · p. 2
• The embeddings support object-level motion transfer and selective extraction and recombination of regional dynamics for controllable scene animation.Source: §1 · p. 2

• Text prompts provide only coarse, indirect guidance over object motion, motivating object-centric control for individual entities.Source: §2 · pp. 2–3
• Trajectory, flow, and other image-plane controls can be precise but remain coupled to viewpoint, object layout, and scene configuration.Source: §2 · p. 3
• Motion-transfer approaches using global or backbone feature dynamics often lack explicit object-level structure, limiting control over where motion is extracted and assigned.Source: §2 · p. 3
• Unlike trajectory-based controls and globally represented motion transfer, this work conditions localized motion representations on spatial regions while preserving full-scene context.Source: §2 · p. 3

3.1 Motivation: Localized Motion Requires Global Context

• Spatial crops can make object motion ambiguous with camera motion because they remove the global reference frame and interaction context.Source: §3.1 · p. 3

3 Method

• The framework combines region-specific motion tokens, appearance-grounding content tokens, and masked trajectory supervision.Figure 2 depicts the motion encoder, content encoder, and masked flow decoder.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2 Promptable Localized Motion Embeddings

• The motion encoder processes the full video and uses region-specific content queries to produce persistent motion tokens grounded in scene context.The encoder can use cues such as camera motion, occlusions, and object interactions.Source: §3.2 · p. 4
• A time-invariant content token grounds each queried region's appearance and helps specify which entity's motion to extract.Source: §3.2 · p. 4
• The masked tracks decoder predicts future positions of points sampled within each entity's region, supervising localized motion tokens.Only valid future track positions within a region are evaluated.Source: §3.2 · pp. 4–5

3.3 Localized Motion-Guided Video Generation

• For motion-guided generation, region motion tokens are paired with content tokens and appended to the video-token sequence processed by self-attention.Source: §3.3 · p. 5, Figure 3 · p. 5
• The generation backbone is fine-tuned with LoRA to incorporate the added conditioning while preserving pretrained weights.Source: §3.3 · p. 5

4.1 Motion Transfer

• Participants preferred our approach to the next-best baseline by at least 27.78% in realism and 9.8% in motion transfer quality.The blinded user study assessed realism, transferred-motion quality and locality, and visible artifacts.Source: §4.1 · p. 7, Table 2 · p. 9
• Our method achieves the best overall motion selectivity, balancing in-region motion fidelity with reduced out-of-region leakage.It combines localized control with semantic motion transfer, improving in-region fidelity and localization over DisMo.Source: §4.1 · p. 6, Figure 4 · p. 7, Table 1 · p. 8

4.2 Localized Action Classification

• For localized action classification, preserving full-scene context while querying the actor outperforms cropping-based and global approaches; either removing context or localization degrades performance.The evaluation uses actor-mask-conditioned motion embeddings and reports top-1 accuracy and weighted F1.Source: §4.2 · pp. 7–8, Table 4 · p. 10

4.1 Motion Transfer

• In multi-source transfer, our method achieves the best overall motion selectivity and substantially reduces leakage while remaining competitive in motion fidelity.The setting attaches motion from distinct source videos to different target subjects in a shared scene.Source: §4.1 · p. 6, Table 3 · p. 9

4.3 From Local to Global: Generic Motion Representations

• With a full-frame query, the localized model performs strongly across global action benchmarks and achieves the best results on several datasets.This adaptation requires no architectural changes and retains the ability to produce region-specific representations when given a query mask.Source: §4.3 · p. 9, Table 5 · p. 10
• On SemanticMoments appearance-transformation retrieval, our method outperforms all baselines except DisMo, reaching near-parity with it and surpassing appearance-centric representations.The evaluation uses top-1 retrieval on the synthetic split across controlled appearance transformations.Source: §4.3 · p. 10, Table 6 · p. 10
• Under visual distractors, localized LAM achieves the lowest mean NMSE, improving over uncropped DisMo by 32.3%.Direct RGB masking also improves DisMo but remains weaker than the localized representation.Source: §4.3 · pp. 10–11, Table 7 · p. 11

5 Conclusion

• The method conditions motion encoding on spatial masks while processing full video, producing temporally consistent, spatially precise embeddings.The conclusion contrasts this with crops that discard context and post-hoc masking that does not condition encoding on the queried region.Source: §5 · p. 11
• Across motion transfer and localized classification, the approach improves controllability, structured scene composition, and motion understanding over cropped or post-hoc-masked global representations.Source: §5 · p. 11
• The learned representations also show strong performance on global video understanding tasks, including classification and retrieval.Source: §5 · p. 11

A.1 Pretraining details

• The pretraining architecture uses a mask-conditioned content encoder, a motion encoder, and a decoder trained to predict point trajectories.The content encoder aggregates frame embeddings toward the masked entity; the motion encoder produces entity-specific temporal embeddings, and the decoder predicts trajectories relative to a source frame.Source: §A.1 · pp. 16–17
• Pretraining uses approximately 17M clips from OpenVid-1M and a self-collected YouTube dataset.The appendix describes the two datasets as covering high-quality open-domain video and a broad range of real-world scenes and motions.Source: §A.1 · p. 17

A.2 Video Model Finetuning Details

• For video-model finetuning, CogVideoX-5B attention and feedforward layers are adapted with rank-128 LoRA modules.Training uses batch size 16, AdamW, and a constant 5 × 10^-5 learning rate; videos are processed at 25 × 480 × 720 and generated at 8 FPS.Source: §A.2 · p. 17
• Motion and content embeddings are temporally aligned to VAE latent frames, then concatenated per region for multi-region conditioning.Three learned padding tokens and non-overlapping groups of four yield seven conditioning vectors; each region's tokens receive temporal and mask-centroid positional encodings.Source: §A.2 · p. 17

Evaluation Details

• SSv2 and Jester test fine-grained temporal reasoning and localized hand-motion sensitivity, respectively.SSv2 minimizes reliance on background context or object identity; Jester contains over 148,000 short clips.Source: §A.3 · p. 18
• ARID, IARD, Diving48, and A2D evaluate low-light robustness, identity and viewpoint invariance, long-term temporal dynamics, and spatially grounded action understanding.SemanticMoments holds motion constant while varying appearance, object identity, and viewpoint to isolate their effects on representation similarity.Source: §A.3 · p. 18
• Track-conditioned baselines extract CoTracker3 point tracks inside source masks, affine-retarget them to target masks, then condition video generation on the tracks and target image.Retargeting aligns mask centroids and uniformly scales tracks to fit the target bounding box.Source: §A.3 · p. 18

B Ablations

• The ablations study prediction distance during trajectory decoding, motion-embedding bottleneck dimensionality, and content-encoder conditioning during trajectory decoding.These design choices are assessed for their effects on learned motion representations.Source: B Ablations · p. 18
• Prediction distance is varied to assess its role in trajectory decoding.The section identifies prediction distance as one of the three investigated aspects.Source: B Ablations · p. 18
• Motion bottleneck dimensionality and content-encoder conditioning are the other two investigated design aspects.The conditioning question concerns trajectory decoding.Source: B Ablations · p. 18

B.1 Prediction Distance

• Prediction distance p sets how many future steps trajectories are predicted from their source-frame starting positions, from sparse flow at p = 1 to full-video prediction at p = 7.Seven variants were evaluated on six action-classification benchmarks.Source: §B.1 · p. 18
• Action-classification results remain similar across prediction distances, with moderate drops on SSv2 and Jester beyond p = 3.The authors adopt p = 3 as a practical default; nearby values yield comparable performance.Source: §B.1 · p. 18, Figure B.1 · p. 19

B.2 Motion Bottleneck Size

• 6× smaller bottlenecks, from d = 768 to d = 128, retain remarkably stable performance across all benchmarks.The result indicates that motion dynamics can be compressed into compact embeddings without significant information loss.Source: §B.2 · p. 19

B.3 Appearance Entanglement Analysis

• Providing the content token during decoding further lowers actor-identity classification accuracy, moving motion embeddings closer to the independence line.The comparison uses IARD actor-identity classification; the decoder can receive a dedicated appearance pathway through the content token.Source: §B.3 · pp. 19–20
• Frozen-embedding reconstruction probes yield the highest L1 and LPIPS errors and lowest CLIP similarity among the compared representations, indicating less recoverable appearance information.The encoder remains frozen during probe training.Source: §B.3 · p. 20

C.1 Localized Motion Transfer

• Across one-to-one and compositional transfer examples, the method is more robust to pose mismatch, scene-motion entanglement, and incorrect subject localization than prior approaches.Low-level approaches show geometric inconsistencies or artifacts, while global methods may move the wrong subject, affect multiple instances, or hallucinate objects.Source: C Qualitative Examples · p. 20, §C.1 · p. 20

C.2 Scene Composition

• In composite transfer, prior methods misassign motions, omit requested components, or induce degenerate camera motion, while ours more reliably matches motions to subjects.The reported errors indicate difficulty jointly modeling subject-specific motion while preserving a stable global scene.Source: §C.2 · pp. 20–21
• Content-token conditioning markedly reduces actor-identity predictability toward the 20% independence line on IARD.The probe uses a kNN classifier with five actors; 20% accuracy corresponds to perfect independence.Source: Figure B.3 · p. 21
• The appearance reconstruction probe treats lower CLIP similarity and higher L1 and LPIPS errors as evidence of less recoverable appearance information.Source: Table B.3 · p. 21

C.3 Text-prompted Generation

• Reconstructions from the motion embeddings preserve less actor appearance than reconstructions from baseline representations.Source: Figure B.4 · p. 22

C.4 User Study Details

• The blinded Prolific study used 18 participants and 17 non-cherrypicked examples to rate realism, transfer quality and locality, and visible artifacts.Participants saw videos without method or model identity information.Source: §C.4 · p. 22

C.5 Ethical Impact

• The authors identify deceptive video manipulation as a dual-use risk and recommend disclosure, provenance tracking where possible, and safeguards.Source: §C.5 · p. 22