PaperlayerView exact revision on arXiv

Source-linked AI summary

AnimeCeleb: Large-Scale Animation CelebHeads Dataset for Head Reenactment

Kangyeol Kim, Sunghyun Park, Jaeseong Lee, Sunghyo Chung, Junsoo Lee, Jaegul Choo

arXiv:2111.07640v2cs.AIcs.CV

TL;DR

Animation head reenactment is limited by the mismatch between human video datasets and animated appearances. The paper introduces AnimeCeleb, a 3D-model-based dataset and semi-automatic pipeline, plus pose mapping and AniMo for cross-domain motion transfer, and reports high-quality results and superiority over state-of-the-art methods.

  • Problem

    Human head reenactment methods trained on large-scale human video datasets generalize weakly to animation because animated characters have distinct appearances.

  • Method

    The paper constructs AnimeCeleb from controllable 3D animation models and introduces pose mapping with AniMo to transfer human motion to animation heads.

  • Results

    Experiments show that AnimeCeleb enables high-quality animation head reenactment, while AniMo is superior to state-of-the-art methods for cross-domain head reenactment.

  • Takeaways & Limitations

    AnimeCeleb provides a practical resource for developing animation head reenactment models and supports transferring human head motion to animation heads.

Abstract

from arXiv · show

We present a novel Animation CelebHeads dataset (AnimeCeleb) to address an animation head reenactment. Different from previous animation head datasets, we utilize 3D animation models as the controllable image samplers, which can provide a large amount of head images with their corresponding detailed pose annotations. To facilitate a data creation process, we build a semi-automatic pipeline leveraging an open 3D computer graphics software with a developed annotation system. After training with the AnimeCeleb, recent head reenactment models produce high-quality animation head reenactment results, which are not achievable with existing datasets. Furthermore, motivated by metaverse application, we propose a novel pose mapping method and architecture to tackle a cross-domain head reenactment task. During inference, a user can easily transfer one's motion to an arbitrary animation head. Experiments demonstrate the usefulness of the AnimeCeleb to train animation head reenactment models, and the superiority of our cross-domain head reenactment model compared to state-of-the-art methods. Our dataset and code are available at https://github.com/kangyeolk/AnimeCeleb.

1 Introduction

AnimeCeleb addresses weak animation-domain generalization by constructing a large-scale, pose-annotated animation head dataset from controllable 3D models. The paper also introduces pose mapping and AniMo for transferring human motion to animation heads, with experiments showing superior cross-domain reenactment.

  • 1 Introduction: Human-video-trained reenactment methods generalize weakly to animation because animated characters differ substantially from human heads in appearance.The paper identifies explicit lines and large eyes as examples of this domain gap.
  • 1 Introduction: AnimeCeleb uses 3D animation models as controllable image samplers, enabling unlimited multi-pose images with detailed semantic pose vectors and arbitrary-resolution rendering.The dataset construction pipeline combines model collection, semantic annotation, and image rendering.
  • 1 Introduction: The semi-automatic creation process uses an annotation system and Blender-based head detection, pose manipulation, and automatic rendering.The pipeline filters expression-irrelevant morphs before rendering annotated images.
  • 1 Introduction: A pose-mapping method transforms AnimeCeleb pose vectors into 3DMM parameters, allowing AnimeCeleb and VoxCeleb to share pose representations.AniMo uses these shared representations to manipulate animation heads according to motion learned across domains.
  • 1 Introduction: Experiments show that AnimeCeleb trains effective animation head reenactment baselines, while AniMo outperforms state-of-the-art methods on cross-domain head reenactment.The contribution claims both dataset usefulness and cross-domain superiority.

2 Animation CelebHeads Dataset

AnimeCeleb constructs an animation head dataset by rendering controllable 3D animation models with semantically annotated expressions and head rotations. Its pipeline supports pose-conditioned reenactment, while experiments show that dataset consistency affects reenactment quality and that pose-vector training performs strongly.

  • Dataset Creation: The creation pipeline collects models, renders heads, semantically annotates morphs, and samples expression and rotation combinations.Blender supports rendering and pose manipulation, while annotation maps source morphs to unified target semantics.
  • Dataset Creation: AnimeCeleb uses 3D animation models to generate controllable multi-pose head images with corresponding pose annotations.The pipeline samples target morphs and head rotations to compose pose vectors conditioning rendered images.
  • Semantic Annotation: The annotation system defines 23 target morphs spanning eyes, eyebrows, and mouth, with individual inspection correcting group-level matches.These target morphs provide the expression dimensions used in the pose vectors.
  • Dataset Properties: The dataset contains 3,613 3D models, and most models have more than 20 target morphs after annotation.Raw source morph counts average 49, while the annotated collection provides dense usable expression annotations.
  • Animation Head Reenactment: AnimeCeleb-trained reenactment models outperform keypoint-trained PIRenderer and FOMM on AnimeCeleb, whereas iCartoonFace training produces blurry outputs amid high within-identity variation.The quantitative comparison is reported in Table 2, and the iCartoonFace diagnosis is supported by qualitative and inception-score analysis.

3 Cross-Domain Head Reenactment

The paper addresses cross-domain head reenactment by mapping AnimeCeleb poses into a shared 3DMM representation and combining domain-specific networks. AniMo outperforms baselines on cross-domain image quality and head-pose imitation.

  • 3.2 Training Pipeline: Training reconstructs animation and human driving images from within-domain source–driving pairs because direct cross-domain supervision is unavailable.The model uses separate animation and human pairs while sharing the pose-processing design.
  • 3.1 Driving Pose Representations: The pose-mapping function converts AnimeCeleb’s 20-dimensional pose vector into 3DMM expression, rotation, and translation parameters.It maps 17 expression coefficients and three head angles into the shared human pose space, with zero translation appended.
  • 3.2 Training Pipeline: A domain-agnostic motion network encodes mapped animation poses and human 3DMM poses into a common latent pose representation.The resulting code supplies affine parameters for adaptive instance normalization in the warping and editing networks.
  • 3.3 Experiments: Experiments use AnimeCeleb and VoxCeleb, evaluating self-identity and cross-domain reenactment with FID, SSIM, and Head Angle Error.The setup includes 18,503 downloadable VoxCeleb videos for training and 504 for testing.
  • 3.3 Experiments: AniMo outperforms all baselines on cross-domain reenactment for both image quality and head-pose imitation.Table 3 reports quantitative comparisons, while the authors characterize the result as superior pose transfer across domains.
  • 3.3 Experiments: Qualitatively, AniMo preserves source identity and vivid textures while accurately reflecting the driving pose, aided by domain-specific networks.Baselines either leak identity and blur outputs or retain structure with less detailed textures.

4 Conclusions

AnimeCeleb is presented as a large-scale, richly annotated animation-head dataset built from 3D models and a semi-automatic Blender pipeline. The paper also introduces pose mapping and AniMo for transferring human head motion to animation heads.

  • 4 Conclusions: AnimeCeleb uses simulated facial expressions and head rotations from 3D animation models to create organized images with rich pose annotations.A Blender-based semi-automatic pipeline and semantic annotation tool support dataset construction.
  • 4 Conclusions: The proposed pose mapping and AniMo architecture address cross-domain reenactment by transferring human head motion to animation heads.Experiments demonstrate effectiveness for cross-domain reenactment and intuitive image editing.

A Related Work

Prior animation datasets generally lack detailed pose annotations, limiting their direct support for generative head reenactment. AnimeCeleb instead provides grouped same-identity images paired with detailed pose information.

  • A Related Work: Existing animation datasets provide preprocessed heads, but generative modeling remains non-trivial because detailed pose annotations are absent.Accurate pose representations are also difficult to obtain for collected animation images.
  • A Related Work: Head reenactment combines source identity information with motion information from another frame, using representations such as keypoints or 3DMM parameters.Most approaches train with two frames sampled from the same video.
  • A Related Work: AnimeCeleb supplies numerous same-identity image groups with detailed pose annotations for generation tasks.The dataset is positioned as a complement to existing animation-head resources.

B Details of Data Creation Process

The data-creation process annotates animation-model morphs, samples semantic expressions and head angles, and renders paired frontalized and rotated images. Its tools provide visual inspection, mapping records, and pose-sampling procedures.

  • B Details of Data Creation Process: The process begins with annotated target morphs indexed as source morphs and uses their sampled values to construct pose vectors.The pseudocode initializes a pose vector before assigning sampled eye, eyebrow, mouth, roll, pitch, and yaw values.
  • B Details of Data Creation Process: The pipeline documents target-morph visualizations, annotation interfaces, source-to-target mappings, pose sampling, and rendered examples.These components are represented in Figures 9–11, Table 4, and Algorithm 1.
  • B Details of Data Creation Process: The annotation system visualizes manipulated target morphs and matches source morphs to semantic targets for eyes, eyebrows, and mouth.Eye and eyebrow semantics distinguish left-related, right-related, and both-related controls, while head rotation and mouth use single values.
  • B Details of Data Creation Process: Pose sampling randomly selects semantic morphs, samples their values, and draws roll, pitch, and yaw from U(−20°, 20°).The sampled values populate a pose vector containing expression and head-angle entries.
  • B Details of Data Creation Process: Given a 3D animation model, rendering produces frontalized-expression images and rotated-expression images, with four shading styles applied.The latter are created after adding sampled head angles.

C Additional AnimeCeleb Samples and Experimental Results

Additional results show AnimeCeleb supports diverse sampled images, animation head reenactment, colorization, and image harmonization applications.

  • Additional Examples from AnimeCeleb: AnimeCeleb provides additional sampled images spanning multiple characters, pose groups, and rendering styles.The samples include frontalized-expression and rotated-expression groups, with identical pose vectors rendered using different shaders.
  • Other Animation Images Head Reenactment: The PIRenderer trained with AnimeCeleb successfully reenacts poses on animation images from Waifu Labs and Danbooru 2019.The model generates favorable outputs by imitating driving-image head poses.
  • Animation Colorization: AnimeCeleb-trained colorization models produce plausible automatic and reference-guided colorizations for animation sketches.Reference-based colorization also generalizes to reference images collected from online cartoons.
  • Image Harmonization: AnimeCeleb supports image harmonization that blends animation foregrounds with backgrounds and accessories in similar styles.The harmonization results refine backgrounds and accessories around animation characters.

D Implementation Details of the AniMo and Baselines

AniMo combines motion, warping, and editing networks with perceptual, style, and reconstruction objectives, while baselines are implemented for comparison.

  • Motion Network: The motion network maps 3DMM parameters to a latent motion code that controls AdaIN parameters in the warping and editing networks.It uses a four-layer multilayer perceptron and produces z ∈ R256 from m ∈ R70.
  • Warping and Editing Networks: The warping network uses an encoder-decoder with U-Net skip connections and AdaIN to inject motion while estimating optical flow.The editing network propagates multiscale features through a hourglass structure and also applies AdaIN.
  • Objective Functions: A perceptual reconstruction loss trains warping to align a warped source image with the driving image in pretrained VGG-19 feature space.The loss minimizes ℓ1 distances between activation maps rather than pixel values.
  • Objective Functions: The editing network uses reconstruction and style losses, combined with warping losses in the full AniMo objective.The style loss matches Gram-matrix statistics, and the framework is jointly trained on AnimeCeleb and VoxCeleb.
  • Training and Baselines: AniMo is trained in two stages, while comparisons include FOMM, PIRenderer, and LPD under their specified dataset settings.The training uses Adam with an initial learning rate of 0.0001 and batch size 12.

E Additional Head Reenactment Results of AniMo

Additional experiments evaluate AniMo across self-identity, cross-identity, and cross-domain reenactment, plus pose editing and unseen animation images.

  • VoxCeleb Reenactment: On VoxCeleb self-identity reenactment, AniMo produces perceptually realistic outputs comparable to the baselines.The larger performance gap appears for cross-identity inputs, where FOMM often fails to produce photo-realistic results.
  • AnimeCeleb Reenactment: Models trained jointly on AnimeCeleb and VoxCeleb outperform VoxCeleb-only models on AnimeCeleb self-identity and cross-identity reenactment.VoxCeleb-only models perform poorly on AnimeCeleb, while joint training produces promising outputs; FOMM remains limited in texture synthesis.
  • Cross-Domain Head Reenactment: AniMo produces photo-realistic cross-domain reenactment results compared with baselines in both domain-transfer directions.The evaluated directions are VoxCeleb-to-AnimeCeleb and AnimeCeleb-to-VoxCeleb.
  • Intuitive Image Editing: AniMo enables explicit control of head roll, pitch, yaw, and facial expressions for animation and human heads.The qualitative editing results manipulate eyes and mouth as well as head rotation.
  • Other Animation Images: AniMo transfers driving poses to unseen animation images from Waifu Labs, Naver Webtoon, Face Sketches, and 2D Disney samples.The experiments also report issues such as background distortion.
  • User Study and Dataset Comparison: Users report that AnimeCeleb has comparable head-pose and expression diversity to iCartoon, alongside favorable cleanness.Figure 20 compares pose statistics with VoxCeleb and user-study scores with iCartoon.

F Discussions

The discussion identifies scope and quality limitations of AnimeCeleb and AniMo, while outlining concrete directions for expanding the dataset and improving reenactment outputs.

  • F Discussions: The creation protocol currently generates multi-pose but single-view animation head images with limited poses because of budget constraints.The stated expansion directions include more views, resolutions, and lighting conditions.
  • F Discussions: AnimeCeleb is currently limited to 256 × 256, frontal, center-aligned, background-free images, leaving higher-resolution, multi-view, lighting, and varied-background settings for future work.The authors describe renderable 1024 × 1024 images, camera-angle variation, controllable illumination, and inference preprocessing as improvement directions.
  • F Discussions: Cross-domain reenactment can lose detailed eye or mouth poses because VoxCeleb-derived poses may not identify their exact positions in animation heads.The authors plan to incorporate expression-detail correctness during training.
  • F Discussions: AnimeCeleb images are center-aligned and background-free, so inputs with complicated backgrounds can produce background artifacts and degraded pose details.The authors propose shifting source and driving heads or adding inference-time preprocessing.
Loading 2111.07640v2…