PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation

Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, Dahua Lin

arXiv:2407.17438v3cs.CVcs.AIcs.LG

TL;DR

Human image animation lacks accessible high-quality public datasets, and existing approaches often neglect camera motion. HumanVid combines curated real-world and synthetic videos with human- and camera-motion annotations, alongside the CamAnimate baseline. The authors report state-of-the-art performance in controlling human pose and camera motion, particularly for complex motions.

  • Problem

    High-quality training datasets are often inaccessible, and human image animation approaches commonly neglect camera motion.

    Source: §1 · pp. 1–2

  • Method

    HumanVid combines curated Internet videos with synthetic videos and annotations for human and camera motion, and CamAnimate conditions generation on both motions.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    The authors report that CamAnimate trained on HumanVid achieves state-of-the-art performance in controlling human pose and camera motion, particularly for complex motions.

    Source: §5 · p. 10

  • Takeaways & Limitations

    HumanVid provides a benchmark for camera control in human videos and a basis for transparent evaluation.

    Source: §1 · p. 3, §5 · p. 10

    Internet-data pose and camera annotations may contain noise, while the synthetic assets and backgrounds are limited and rendering quality is below professional-camera footage.

    Source: §5 · p. 10

Abstract

from arXiv · show

Human image animation involves generating videos from a character photo, allowing user control and unlocking the potential for video and movie production. While recent approaches yield impressive results using high-quality training data, the inaccessibility of these datasets hampers fair and transparent benchmarking. Moreover, these approaches prioritize 2D human motion and overlook the significance of camera motions in videos, leading to limited control and unstable video generation. To demystify the training data, we present HumanVid, the first large-scale high-quality dataset tailored for human image animation, which combines crafted real-world and synthetic data. For the real-world data, we compile a vast collection of real-world videos from the internet. We developed and applied careful filtering rules to ensure video quality, resulting in a curated collection of 20K high-resolution (1080P) human-centric videos. Human and camera motion annotation is accomplished using a 2D pose estimator and a SLAM-based method. To expand our synthetic dataset, we collected 10K 3D avatar assets and leveraged existing assets of body shapes, skin textures and clothings. Notably, we introduce a rule-based camera trajectory generation method, enabling the synthetic pipeline to incorporate diverse and precise camera motion annotation, which can rarely be found in real-world data. To verify the effectiveness of HumanVid, we establish a baseline model named CamAnimate, short for Camera-controllable Human Animation, that considers both human and camera motions as conditions. Through extensive experimentation, we demonstrate that such simple baseline training on our HumanVid achieves state-of-the-art performance in controlling both human pose and camera motions, setting a new benchmark. Demo, data and code could be found in the project website: https://humanvid.github.io/.

1 Introduction

Human image animation research faces limited access to high-quality public training data and often neglects camera motion. HumanVid combines curated Internet videos with synthetic videos and evaluates the dataset with a baseline that considers both human and camera motion.

1 Introduction

  • Private training datasets and limited accessible alternatives hinder fair evaluation, while pose-focused methods neglect camera movement and struggle with new viewpoints.The cited challenges include accessible datasets’ limits in scale and quality and methods’ reliance on 2D poses from static-camera videos.Source: §1 · pp. 1–2
  • HumanVid combines high-quality Internet videos with synthetic videos to provide human- and camera-motion annotations.The real-world videos are filtered to retain pixel motion resulting exclusively from human and camera motion; the collection contains over 20K human-centric videos at 1080P.Source: §1 · p. 2
  • Its synthetic pipeline uses diverse character assets and rule-based camera trajectories to generate videos with accurate human- and camera-motion annotations.Camera keyframes are sampled at multiple locations, directed toward the face, then connected and smoothed into natural movements.Source: §1 · pp. 2–3
  • The paper presents HumanVid as the first large-scale human image animation dataset and establishes a baseline and evaluation benchmark for the field.The authors report experiments validating dataset components and establishing state-of-the-art performance.Source: §1 · p. 3

2 Related Work

Prior work spans human image animation, camera-aware video generation, and human-video datasets. The paper positions HumanVid as the first to use synthetic data for video generation and to introduce camera control into human-centric video generation.

Human Image Animation

  • Human image animation methods use explicit skeleton guidance and have moved from GANs toward diffusion models, but the cited works mostly use static cameras.Examples include Animate Anyone’s reference network and pose guider, MagicAnimate’s DensePose and ControlNet, and Champ’s SMPL sequence with rendered depth and normal.Source: §2 · p. 3

Camera-aware Video Generation

  • Camera-aware video generation methods range from rough camera control to camera-pose conditioning, but this paper presents itself as the first to introduce camera control into human-centric video generation.The related work describes camera extrinsics, Plücker ray embeddings, and camera embeddings as control approaches.Source: §2 · p. 3

Human Video Datasets

  • Existing human-video datasets include Internet collections and synthetic datasets designed mainly for pose estimation or multi-view reconstruction, rather than video generation.The Internet examples are TikTok and UBC-Fashion; the synthetic examples include SURREAL, AGORA, HSPACE, GTA-Human, SynBody, and BEDLAM.Source: §2 · pp. 3–4

3 Dataset

HumanVid combines curated Internet videos with synthetic human videos, providing motion annotations and a scalable rendering pipeline. Its dataset construction spans diverse characters, retargeted motion, varied scenes and camera trajectories, Internet-data curation, and the CamAnimate baseline.

3 Dataset

  • HumanVid combines scalable synthetic videos with copyright-free Internet videos, using pose estimation for automatic annotation without human supervision.Synthetic videos are rendered with UE5 or Blender; the Internet-video component adds human-centric real-world appearances.Source: §3 · p. 4

3.1 Synthetic Data Construction

  • Synthetic data construction uses character creation, motion retargeting, and 3D scene and camera placement as its three steps.The pipeline renders one character moving through varied 3D scenes and camera trajectories.Source: §3.1 · p. 4

3.1.1 Character Creation

  • Characters span SMPL-X humans and anime assets, with varied body shapes, skin textures, clothing, and 10K manually selected VRoidHub models.The human-like assets include 271 body shapes, 100 skin textures across seven ethnic groups, and 111 outfits with 1,691 clothing textures.Source: §3.1.1 · pp. 4–5

3.1.2 Motion Retargeting

  • Motion-capture and open-source motions are retargeted to SMPL-X and anime characters, with clothing and hair motion following the anime characters’ source motions.SMPL-X motion sampling uses motion annotations; anime assets use retargeting software to accommodate diverse skeleton lengths.Source: §3.1.2 · p. 5

3.1.3 3D Scenes and Camera Placements

  • Synthetic scenes use detailed indoor and outdoor backgrounds, while rule-based 6-DoF camera trajectories sample keyframes and smoothly interpolate camera movement around the person.Keyframes sample camera locations 3–5 m away and 0.6–1.2 m high in front of the human, with roll between −30° and 30°; rendering supplies camera, pose, segmentation, depth, and normal annotations.Source: §3.1.3 · p. 5

3.2 Internet Data Curation

  • The Internet-data pipeline filters videos for human-centric motion and uses SLAM-based estimation to recover camera trajectories, treating unreconstructable low-texture backgrounds as static.It collects about 20K clips and filters out videos with very large camera movements or sudden shot changes.Source: §3.2 · p. 6
  • Over 20K Internet clips and 10M frames at 1080P make the real-world dataset larger and higher-resolution than TikTok, UBC-Fashion, and IDEA-400; the synthetic dataset is 5× Bedlam’s scale.The synthetic dataset also provides camera and human-pose ground truth and more diverse camera trajectories.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 CamAnimate

  • CamAnimate combines CameraCtrl camera-pose control with Animate Anyone’s character-animation framework, encoding camera trajectories while separate modules support appearance consistency and pose control.It is trained end-to-end on videos with both human and camera movement and can also generate static-camera videos.Source: §3.3 · pp. 6–7, Figure 4 · p. 7

4 Experiments

The experiments compare CamAnimate with prior methods under static- and moving-camera settings, then examine user preferences, dataset and method components, and camera-control evaluation limits.

4 Experiments

  • The benchmark uses the middle frame as reference, predicts 24 frames, and evaluates with PSNR, SSIM, LPIPS, FID, and FVD; reconstruction metrics are unreliable for moving-camera videos.The moving-camera test sets include 40 portrait and 40 landscape videos from collected Internet videos.Source: §4 · p. 7

4.1 Comparison with the State-of-the-Art

  • Animate Anyone shows inaccurate human appearance, Champ misses 3D skeletons, and Magic-animate misses facial expressions; CamAnimate generates expressions, poses, and camera-corresponding background motion.The qualitative comparison highlights artifacts with red boxes.Source: §4.1 · pp. 8–9, Figure 5 · p. 8
  • CamAnimate achieves the best performance in almost all metrics on static-camera TikTok and UBC-Fashion, while prior methods struggle to generate natural videos with camera movement.The comparisons cover static-camera datasets and collected human videos with camera movements.Source: Table 3 · p. 8, §4.1 · p. 8, Table 4 · p. 9
  • In in-the-wild examples, CamAnimate generates visually pleasant videos with explicit camera movements that the authors describe as close to professionally filmed movies.Figure 6 presents the in-the-wild results.Source: §4.1 · p. 9, Figure 6 · p. 9

4.2 Ablation Study

  • On HumanVid’s predominantly moving-camera videos, the same-training-set variant without camera conditioning differs significantly from CamAnimate, indicating the camera encoder is necessary in this setting.Table 6 compares CamAnimate with original Animate Anyone trained without camera condition.Source: §4.2 · p. 9, Table 6 · p. 10

4.1 Comparison with the State-of-the-Art

  • The user study reports 0.44 points and 0.73 top-1 preference for CamAnimate, a dominant advantage over the compared methods.The study used 20 participants ranking results on 10 videos.Source: §4.1 · p. 8, Table 5 · p. 10

4.2 Ablation Study

  • The final two-stage training strategy outperforms both Internet-only training and training on both video types in both stages.Internet videos support appearance modeling, while synthetic videos improve camera control accuracy and overall visual quality.Source: §4.2 · pp. 9–10, Table 7 · p. 10
  • Quantitative camera-trajectory evaluation is deferred because structure-from-motion methods fail when both people and cameras move, making trajectory metrics inaccurate.The authors leave quantitative verification of camera control to future work.Source: §4.2 · p. 10

5 Conclusion

The paper concludes that combining real-world and synthetic data supports human animation with complex human and camera motions. It also identifies data and rendering limitations, alongside possible deepfake misuse.

5 Conclusion

  • The combined real-world and synthetic dataset establishes a camera-control benchmark, and the simple baseline shows superior performance particularly with complex human and camera motions.The authors intend the dataset to support transparent and comprehensive evaluation.Source: §5 · p. 10
  • Internet-data annotations may contain pose or camera noise, while synthetic assets and scenes are limited and render below professional-camera footage.These limitations qualify the dataset’s annotation and synthetic-data scope.Source: §5 · p. 10
  • More capable generative models could produce realistic deepfakes that may be misused to spread misinformation.Source: §5 · p. 10

Checklist

The checklist reports coverage of asset sourcing, research ethics, and reproducibility details, while noting that experimental error bars are absent.

Checklist

  • The paper reports discussing work limitations and potential negative societal impacts, and refers readers to supplementary material for consent and personally identifiable or offensive content.Source: Checklist · p. 16
  • The checklist reports that code, data, and reproduction instructions, training details, and compute resources are provided in the paper or supplementary material.Source: Checklist · p. 16
  • The authors do not report error bars for PSNR, SSIM, LPIPS, FID, or FVD because of extensive computation costs.Source: Checklist · p. 16
  • The checklist reports citing creators of existing assets and discussing asset licenses in the supplementary material.Source: Checklist · p. 16

A.1 Implementation Details of CamAnimate

CamAnimate uses a two-stage training setup with pretrained components and camera embeddings derived from camera parameters.

A.1 Implementation Details of CamAnimate

  • Training runs for 30,000 iterations in stage one and 10,000 in stage two, using a learning rate of 1e-5 and AdamW.The default setup uses batch size 8 in stage one and batch size 1 in stage two, with eight NVIDIA A100 GPUs for training.Source: §A.1 · p. 17
  • Stage two freezes the denoising UNet, reference UNet, and pose guider, training only the camera encoder and motion module.The motion module is initialized with AnimateDiff V3 weights.Source: §A.1 · p. 17
  • The camera embedding uses CameraCtrl’s Plücker representation computed from intrinsic and extrinsic parameters.For real internet data, intrinsics use a heuristic and extrinsics come from SLAM; synthetic data exports both parameter types directly.Source: §A.1 · p. 17
  • The Denoising UNet and ReferenceNet start from Stable Diffusion 1.5, while the Pose Guider starts from ControlNet weights trained on OpenPose.Training mixes horizontal and vertical videos at resolutions of 896×512 and 512×896, respectively.Source: §A.1 · p. 17

A.2 More Details about Rendering of Synthetic Data

This appendix points readers to flow-chart and illustration material describing the synthetic-data rendering pipeline and anime-character creation.

A.2 More Details about Rendering of Synthetic Data

  • Figure 7 provides a flow chart of the synthetic data creation pipeline and rendering process.The text refers readers to the figure for a detailed visualization of that process.Source: §A.2 · p. 17, Figure 7 · p. 17
  • Figure 8 illustrates the video-rendering process used to create anime characters.Source: §A.2 · p. 17

A.3 Camera Movement Comparison

The synthetic camera trajectories span larger frame-wise offsets than the real-data trajectories, and the rendering pipeline can also provide human masks, depth, and normal maps, though these are not included in the dataset.

A.3 Camera Movement Comparison

  • Synthetic data has a larger distribution of frame-wise camera pose offsets than real data, and models trained on these trajectories follow real-world camera trajectories well in inference.The comparison covers six dimensions: XYZ location and roll, yaw, and pitch; the synthetic trajectories use two keyframes, with three- and five-keyframe options also possible.Source: §A.3 · pp. 17–18, Figure 9 · p. 18

A.4 Visualization of Human Mask, Depth and Normal

  • The rendering pipeline produces paired human segmentation masks, depth maps, and normal maps, but HumanVid does not provide these outputs.The example renders SMPL-X characters placed in a city scene, with the scene and character jointly rendered in Unreal Engine.Source: §A.4 · p. 18

A.5 Visualization of Human Keypoints Tracking

The rendering pipeline focuses on human pose and camera parameters but can also track human keypoints; the appendix separately describes how participants ranked animation outputs in a user study.

A.5 Visualization of Human Keypoints Tracking

  • The rendering pipeline can track human keypoints, though extending it to generate point-tracking data requires additional development.The example tracks human keypoints rather than all scene or object points.Source: §A.5 · p. 18, Figure 11 · p. 19

A.6 Details of User Study Questionnaires.

  • The user study tested 10 samples, with participants ranking four animation methods by character appearance and motion naturalness, background smoothness during camera movement, and image stability with minimal color flickering.The listed criteria are ordered by descending importance.Source: §A.6 · p. 19
  • Videos were randomized in order for each user-study question, which presented the first-frame image alongside the four generated videos.Source: §A.6 · p. 19

B Dataset Documentation

HumanVid’s documentation outlines the data preparation and two-stage CamAnimate training workflow, and specifies how camera parameters are represented. The dataset also supports static-camera animation by passing parameters for a static camera.

B Dataset Documentation

  • Static-camera human image animation is supported by passing camera parameters that represent a static camera to the trained model.The dataset is designed for training camera-controllable human image animation models.Source: B Dataset Documentation · p. 20
  • Training preparation combines downloaded internet and synthetic videos with human poses, internet-video camera trajectories, and metadata compiling valid parameters and videos.The documented steps extract poses with DWPose and internet camera trajectories with Droid-SLAM, masking humans before SLAM.Source: B Dataset Documentation · p. 20
  • CamAnimate training uses a first stage for image-level appearance and pose control, followed by a second stage for video-level motion representations and camera control.The first stage also trains the image-level camera encoder.Source: B Dataset Documentation · p. 20
  • Camera records use timestamp, position, and quaternion fields; internet parameters are camera-to-world, while synthetic parameters are world-to-camera and include two normalized focal-length values.The documented workflow also calls for pretrained checkpoints for CLIP, SD 1.5, Animatediff, and CameraCtrl.Source: B Dataset Documentation · p. 20

C Author Statements

The author statements specify HumanVid’s CC-BY 4.0 license and list licenses or permissions for assets and video sources. The maintenance plan distinguishes cloud-hosted synthetic videos from internet videos distributed through download links.

C Author Statements

  • HumanVid is released under CC-BY 4.0, and the authors state they bear responsibility in case of rights violations.Source: C Author Statements · p. 20

C.1 Licenses

  • The listed sources include BEDLAM assets for SMPL-X character videos, Pexels internet videos, permitted VroidHub avatars, and the Rokoko motion-retargeting addon.The passage describes BEDLAM assets as available for academic usage; VroidHub assets used have individual permissions, and the Rokoko license grants use in the dataset.Source: §C.1 · pp. 20–21

D Maintenance Plan

  • Synthetic videos will be maintained on a cloud drive, while internet videos will be made available through links because the raw videos cannot be redistributed.Source: D Maintenance Plan · p. 21
Loading 2407.17438v3…