PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation

Yingjie Chen, Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo

arXiv:2501.05020v2cs.CV

TL;DR

Collaborative camera and object motion control remains difficult because existing representations can conflict or lack fine granularity. Perception-as-Control builds a 3D-aware scene representation and uses its rendered perceptions as motion controls; the paper reports consistently improved controllability across multiple benchmarks. Its performance remains constrained by the base model, particularly for human motion.

  • Problem

    Existing motion representations still struggle to support collaborative camera and object control without conflicts and with fine control granularity.

    Source: §1 · p. 2

  • Method

    Perception-as-Control represents key object parts and the surrounding world in a simplified 3D-aware scene, using rendered perceptions as motion controls.

    Source: §1 · pp. 2–3

  • Results

    The authors report that the 3D-aware representation improves motion controllability consistently across multiple benchmarks and is orthogonal to model architectures.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework supports motion generation, cloning, transfer, and editing by translating user intentions into collaborative motion control.

    Source: §1 · p. 3

    The approach is constrained by base-model performance and does not handle human-related motions satisfactorily.

    Source: E. Limitations and Future Work · p. 15

Abstract

from arXiv · show

Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via various motion representations, while they still struggle to support collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user instructions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive and consistent visual changes. Then, our framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed approach. For more details and qualitative results, please refer to our anonymous project webpage: https://chen-yingjie.github.io/projects/Perception-as-Control.

1. Introduction

Fine-grained collaborative control of camera and object motion remains difficult because existing representations can produce conflicting controls or lack sufficient granularity. The paper introduces a 3D-aware representation and an image-animation framework that use spatially aligned visual changes as motion controls.

1. Introduction

  • Existing methods for camera or object motion control still struggle with conflicts and insufficient granularity when controlling both collaboratively.The paper frames fine-grained collaborative control as a nontrivial problem despite progress in motion representations.Source: §1 · p. 2
  • A useful representation should combine 3D awareness, spatial consistency, fine-grained object-motion flexibility, and alignment with generated frames.The paper contrasts flexible but conflict-prone pixel-offset representations with spatially consistent but less flexible reconstruction-based representations.Source: §1 · p. 2
  • The 3D-aware representation simplifies a scene into unit spheres for key object parts and a world envelope, then renders it from user-intended viewpoints.The spheres represent mostly moving parts and a few static parts; rendered perceptions reflect camera and object motion and spatial relations.Source: §1 · p. 2, Figure 2 · p. 2
  • The framework uses perception results as spatially aligned motion controls, separately encodes camera and object signals, and combines them for diffusion-based animation.The paper also introduces a data-curation pipeline and a three-stage training strategy to balance camera and object control.Source: §1 · pp. 2–3
  • The approach is presented as architecture-orthogonal and supports motion generation, cloning, transfer, and editing through translated user intentions.Its stated aim is precise and harmonious collaborative motion control.Source: §1 · p. 3

2. Related Work

Prior work represents camera motion with camera parameters, Plücker embeddings, reconstructed point clouds, or optical flow, while object-motion methods commonly begin with user-drawn point trajectories. The paper positions its 3D-aware representation as an alternative for fine-grained collaborative control.

2. Related Work

  • Camera-control methods use intrinsic and extrinsic parameters, Plücker embeddings, rendered point clouds, or motion-specific training data and LoRA models.TrajectoryAttention uses dense pixel trajectories for camera control but is described as hard to extend to object motion.Source: §2 · p. 3
  • Object-control methods commonly convert sparse point trajectories into optical flow, trajectory maps, or reorganized pixel features.The cited approaches are reported to have difficulty with simultaneous local object and global camera motion or shape variation.Source: §2 · p. 3
  • Among collaborative approaches, the paper contrasts heterogeneous optical-flow controls, dense point embeddings, and mask-limited object control with its fine-grained 3D-aware representation.The cited methods are AnimateAnything, MotionPrompting, and I2VControl, respectively.Source: §2 · p. 3

3. Perception-as-Control

The section presents a pipeline that builds 3D-aware motion representations from videos and uses them to support fine-grained camera and object control. Its animation framework combines these controls with a reference image and uses staged training to balance the two motion types.

3.1. Preliminaries

  • Video diffusion models temporal dependencies by adding Gaussian noise to video latents and learning reverse transitions, optionally conditioned on reference images and motion controls.The loss trains a parameterized network to predict the distribution of the preceding noisy latent.Source: §3.1 · pp. 3–4

3.2. 3D-aware Motion Representation

  • The representation combines tracked 3D key-part locations with coarse camera poses, then renders object and camera control signals as separate spatial layers.The curation pipeline filters video clips for motion, tracks key parts with SpaTracker, and estimates camera trajectories with TartanVO.Source: §3.2 · pp. 4–5

3.3. Network Architecture

  • A denoising U-Net uses spatially aligned motion controls, with separate camera and object encoders fused together and a ReferenceNet preserving reference-image appearance.Motion modules model temporal information; the reference image is injected through ReferenceNet.Source: §3.3 · p. 5

3.4. Training Strategy

  • Training proceeds from camera-only clips to collaborative camera-and-object clips, then fine-tunes with sparse sphere controls to learn adaptive control ranges.The final stage samples 1–16 spheres from salient or high-trajectory locations; the fusion module and motion modules are trained in that stage.Source: §3.4 · pp. 5–6

4. Experiments

Experiments compare Perception-as-Control with prior methods, test its motion representation and training strategy, and demonstrate fine-grained control and several motion-related applications.

4.1. Experimental Settings

  • Training uses RealEstate-10K and WebVid-10M, with around 61K and 352K curated training clips, respectively, and 1K test clips from each dataset.Evaluation uses visual-quality, object-motion, and camera-motion metrics; the base model is SD1.5.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.2. Comparison with State-of-the-Art Methods

  • Perception-as-Control achieves the best camera and object motion control, while CameraCtrl and MotionCtrl show limitations on large or out-of-distribution camera motions.Optical-flow methods also face conflicting controls and struggle to preserve object appearance under large motions.Source: §4.2 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7

4.3. Ablation Study

  • Ablations indicate that 2D offsets, 2D positions without depth, and 3D unit spheres alone each have drawbacks, while the three-stage strategy supports easier training and harmonious collaborative control.The representation ablation is reported in Table 3 and the training-strategy ablation in Table 4.Source: Table 3 · p. 6, Table 4 · p. 7, §4.3 · p. 8

4.4. Fine-grained Collaborative Motion Control

  • Qualitative examples show control of individual balloons, adaptive control ranges for bamboo unit spheres, and combined camera and object motions.The examples visualize projected unit-sphere centroids, with colors indicating movement direction.Source: §4.4 · p. 8

4.5. Applications

  • The framework demonstrates motion generation, motion cloning, motion transfer across differently scaled and located objects, and region-based motion editing.The examples include user-directed trajectories, source-video motions, semantic-point trajectories, and viewpoint changes.Source: §4.5 · p. 8, Figure 8 · p. 8

5. Conclusion

The paper targets precise, flexible fine-grained collaborative motion control and proposes a 3D-aware representation and image-animation framework. Experiments report consistent controllability improvements across multiple benchmarks and support for varied motion-related applications.

5. Conclusion

  • The paper targets precise, flexible fine-grained collaborative control, beyond prior methods' coarse-grained or inflexible collaborative control.It addresses this by proposing a 3D-aware motion representation that turns camera and object motion into unified, consistent visual changes.Source: §5 · p. 8
  • The framework uses motion-control signals derived from the 3D-aware representation to support varied motion-related applications.The representation is described as orthogonal to model architectures.Source: §5 · p. 8
  • Experiments report consistent motion-controllability improvements across multiple benchmarks.Source: §5 · p. 8

A. More Results of Perception-as-Control

Perception-as-Control supports precise camera control, adaptive multi-instance object control, and harmonious collaborative control.

A.1. Camera-only Motion Control

Camera-only results show adherence to designated trajectories, stable spatial structure, and perspective effects across focal lengths.

  • The framework generates videos that adhere to any designated camera trajectory, including user-provided trajectories and combinations of basic movements.Source: §A.1 · p. 11
  • The world envelope helps generated videos maintain stable, consistent spatial structures while representing camera-movement direction and magnitude.Source: §A.1 · p. 11
  • Unit spheres on essential object parts provide depth awareness and accurate perspective effects under different focal lengths.Source: §A.1 · p. 11

A.2. Object-only Motion Control

The results cover adaptive object-only control, collaborative camera–object control, and integration of the representation with other base architectures.

A.2. Object-only Motion Control

  • Increasing numbers of hot air balloons are controlled individually, while additional bamboo controls adaptively set each unit sphere’s control range for harmonious results.Source: §A.2 · p. 11

A.3. Collaborative Motion Control

  • Collaborative control handles large and delicate object motions while generated videos follow specified camera motion and adapted object motions.Source: §A.3 · p. 11

B. More Details of ++ Versions

  • The representation is incorporated into SVD versions of CameraCtrl and MotionCtrl without major architectural changes, supporting fine-grained collaborative control.Source: B. More Details of ++ Versions · p. 11

C. More Details of Ablation Study

The ablation study distinguishes training strategies, and the applications section describes four ways to translate user intentions into motion-related tasks.

C. More Details of Ablation Study

D. More Details of Applications

E. Limitations and Future Work

The framework has limitations in representing object rotation, generating human motion, and estimating its 3D-aware representations. The authors point to more expressive motion representations, stronger base models, and more precise visual SLAM as future directions.

E. Limitations and Future Work

  • Unit-sphere object parts poorly represent rotation, so the illustrated billiards translate without rotating.The authors suggest exploring more powerful alternatives to express more comprehensive movement.Source: E. Limitations and Future Work · pp. 14–15
  • The approach is constrained by base-model performance and does not handle human-related motions satisfactorily.In the illustrated example, a boy moves forward without reasonable stepping movements.Source: E. Limitations and Future Work · p. 15
  • Off-the-shelf algorithms used to construct the 3D-aware representations affect performance; visual SLAM is suggested for more precise estimation.Source: E. Limitations and Future Work · p. 15
Loading 2501.05020v2…