Source-linked AI summary
Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, Shenghua Gao
TL;DR
Existing human synthesis methods use task-specific pipelines and 2D conditions that do not capture personalized shape, limb rotations, or all source details. The paper combines 3D body mesh recovery with Liquid Warping GANs to unify three tasks, reporting effective results including preserved identity and clothing details, while using LPIPS results updated to the original definition.
Problem
Prior methods use separate pipelines and mainly 2D conditions, limiting representation of personalized body shape, limb rotations, source details, and multiple sources.
Method
The framework combines 3D body mesh recovery with Liquid Warping Blocks that propagate source information in image and feature spaces and support multiple-source warping.
Results
The unified model handles motion imitation, appearance transfer, and novel view synthesis, preserving face identity and clothing details, including under self-occlusion and out-of-domain references.
Takeaways & Limitations
One model trained on one task can be applied to the other two tasks without training from scratch.
Takeaways & Limitations
LPIPS values were updated because an earlier table used 1 − distance rather than the original distance definition.
Abstract
from arXiv · showhide
We tackle the human motion imitation, appearance transfer, and novel view synthesis within a unified framework, which means that the model once being trained can be used to handle all these tasks. The existing task-specific methods mainly use 2D keypoints (pose) to estimate the human body structure. However, they only expresses the position information with no abilities to characterize the personalized shape of the individual person and model the limbs rotations. In this paper, we propose to use a 3D body mesh recovery module to disentangle the pose and shape, which can not only model the joint location and rotation but also characterize the personalized body shape. To preserve the source information, such as texture, style, color, and face identity, we propose a Liquid Warping GAN with Liquid Warping Block (LWB) that propagates the source information in both image and feature spaces, and synthesizes an image with respect to the reference. Specifically, the source features are extracted by a denoising convolutional auto-encoder for characterizing the source identity well. Furthermore, our proposed method is able to support a more flexible warping from multiple sources. In addition, we build a new dataset, namely Impersonator (iPER) dataset, for the evaluation of human motion imitation, appearance transfer, and novel view synthesis. Extensive experiments demonstrate the effectiveness of our method in several aspects, such as robustness in occlusion case and preserving face identity, shape consistency and clothes details. All codes and datasets are available on https://svip-lab.github.io/project/impersonator.html
1. Introduction
Human image synthesis covers motion imitation, appearance transfer, and novel view synthesis, but prior task-specific methods struggle with source detail, body geometry, and multiple sources. The paper addresses these issues with 3D body modeling and Liquid Warping Blocks in a unified framework.
- Human image synthesis includes motion imitation, appearance transfer, and novel view synthesis, with applications in re-enactment, animation, virtual try-on, and media production.
- Prior methods use separate task-specific pipelines, making extension to other synthesis tasks difficult.
- Existing approaches struggle to preserve clothing texture, style, color, and face identity while handling large geometric changes and multiple source inputs.
- The Liquid Warping Block extracts source-preserving features, blends local features into a global stream, and supports warping different parts from multiple sources.
- The framework combines LWB with a 3D parametric model so one trained model handles motion imitation, appearance transfer, and novel view synthesis.
2. Related Work
Related work addresses human synthesis through task-specific generative, graphics-based, and correspondence-based approaches. These methods differ in conditioning and representation, but generally target pose transfer, clothing transfer, or viewpoint changes separately.
- Human Motion Imitation: Human motion imitation methods commonly use conditional GANs or variational auto-encoders to combine images with source or target pose conditions.
- Human Motion Imitation: Early motion-imitation methods concatenate source images and pose conditions, while later approaches generate foregrounds or body parts separately and use dense correspondences.
- Human Appearance Transfer: Graphics-based appearance transfer estimates detailed clothed 3D meshes but requires costly equipment and controlled environments.
- Human Appearance Transfer: Learning-based appearance transfer combines clothing segmentation, source texture features, geometric shape models, and inferred invisible textures.
- Human Novel View Synthesis: Novel view synthesis typically fits correspondence or appearance flow from observable views to novel views and infers invisible textures.
3. Method
The method uses a three-stage Liquid Warping GAN: recover 3D body structure, compose a transformation flow, then synthesize images with linked generator streams. Training is shared across the supported tasks.
- The Liquid Warping GAN consists of body mesh recovery, flow composition, and a GAN module containing the Liquid Warping Block.
- Liquid Warping GAN: The GAN uses background, source-identity, and transfer streams, with LWB propagating source features from source reconstruction into target synthesis.
- The same training pipeline supports motion imitation, appearance transfer, and novel view synthesis after training on one task.
- Body Mesh Recovery: Body mesh recovery estimates 3D pose, shape, camera parameters, and meshes for source and reference images.
- Body Mesh Recovery: The HMR estimator predicts SMPL pose θ ∈R72, shape β ∈R10, and weak-perspective camera K ∈R3 from image features.
3.2. Flow Composition Module
Flow composition renders source and reference mesh correspondences, computes a transformation flow, separates foreground and background, and warps the source image toward the reference condition.
- The module renders correspondence maps for source and reference meshes under the source camera view.
- It projects source mesh vertices into image space and derives face coordinates for correspondence matching.
- Liquid Warping Block: Liquid Warping Blocks aggregate target-stream features with source features warped by bilinear sampling according to transformation flows.
- The transformation flow T ∈RH×W×2 is calculated by matching source and reference correspondence maps with mesh-face coordinates.
- The source image is separated into a front foreground and masked background before being warped into Isyn.
3.3. Liquid Warping GAN
The generator uses separate background, source-identity, and transfer streams, linked by a Liquid Warping Block that preserves source details while synthesizing the target image. The block also supports independently warping and aggregating features from multiple sources.
- The three-stream generator synthesizes the background, reconstructs source identity, and produces the final transferred image.GBG generates the background; GSID reconstructs the source front image; GTSF synthesizes the final result.
- GSID uses a denoising convolutional auto-encoder to extract source features that preserve identity information.The source identity stream guides feature extraction for preserving texture, style, and color.
- The Liquid Warping Block blends source features from GSID into the transfer stream to preserve source information during synthesis.It links source and target streams by fusing source features into GTSF.
- Multiple-source warping independently transforms and aggregates features from different sources into the transfer stream.The framework can combine parts such as a head, upper garment, and lower garment from different source people.
- The background, identity, and transfer streams use ResUnet architectures, while the discriminator follows Pix2Pix.GBG directly regresses the background; GSID and GTSF generate attention and color maps for image synthesis.
3.4. Training Details and Loss Functions
Training samples source-reference image pairs and optimizes perceptual, face identity, attention regularization, and adversarial losses. A model trained for motion imitation can then be applied to appearance transfer and novel view synthesis without training from scratch.
- The unified model randomly samples two images from each video as source and reference, then transfers across the three supported tasks.The authors train on motion imitation and apply the trained model to appearance transfer and novel view synthesis.
- The generator objective combines perceptual, face identity, attention regularization, and adversarial losses.The weights λp, λf, and λa correspond to perceptual, face identity, and attention losses.
- Perceptual loss brings reconstructed source and generated target images closer to their ground-truth images in VGG-19 feature space.The formulation uses a pre-trained VGG-19 network.
- Face identity loss makes the synthesized target face similar to the ground-truth reference face.It regularizes cropped synthesized and ground-truth faces using a pre-trained SphereFaceNet.
- The adversarial loss encourages synthesized images to match the distribution of real images using a conditioned discriminator.The discriminator receives the generated image and target correspondence map.
- Attention regularization keeps the attention map smooth, prevents saturation, and aligns it with silhouettes rendered from the 3D body mesh.Total variation regularization compensates for silhouettes that omit clothes and hair.
3.5. Inference
At inference, the trained model reuses Body Mesh Recovery and Liquid Warping GAN modules across motion imitation, appearance transfer, and novel view synthesis, while task-specific transformation flows vary. These flows use 3D body parameters and mesh correspondences to generate task-specific results.
- Unified inference: The same trained model supports motion imitation, appearance transfer, and novel view synthesis by changing only transformation-flow computation.Body Mesh Recovery and Liquid Warping GAN modules remain unchanged across tasks.
- Motion imitation: Motion imitation transfers the reference pose onto the source shape before rendering source-to-target mesh correspondences.The target SMPL mesh is formed as M(θr, βs), combining reference pose with source shape.
- Novel view synthesis: Novel view synthesis constructs a mesh under a new camera rotation and translation, then computes a transformation flow under the weak-perspective camera.The source mesh is transformed by the novel-view rotation R and translation t.
- Appearance transfer: Appearance transfer splits the flow so source head identity is retained while reference body or clothing information is transferred.The source flow is derived from the projected head silhouette, while the reference-body flow uses mesh correspondences.
- Evaluation setup: The iPER dataset evaluates the three tasks using diverse subjects, body conditions, clothing, and action videos.The dataset includes 30 subjects and 103 clothes in total.
4.1. Evaluation of Human Motion Imitation.
On iPER motion imitation, the method is compared with PG2, SHUP, and DSC using an occlusion-aware evaluation protocol and visual analysis. The reported comparisons indicate better preservation of source shape, identity, and details, while the updated LPIPS values use the original distance definition.
- Evaluation protocol: The evaluation selects frontal, sideway, and occlusive source images to test performance under different degrees of missing information.The frontal image contains the most information, while sideway and occlusive images introduce progressively greater ambiguity.
- Comparison with other methods: Compared with PG2, SHUP, and DSC, the method keeps the source body shape unchanged during motion imitation.The cited example contrasts a tall source person imitating a shorter person’s motion.
- Comparison with other methods: The method generates more realistic ambiguous or invisible content under source self-occlusion than the compared methods.The comparison specifically discusses an image with an invisible face.
- Comparison with other methods: The method better preserves source face identity and clothing details than the compared methods in the reported visual comparisons.Figure 6 highlights these details with colored rectangles.
- Metric clarification: The reported LPIPS values were updated to follow the original distance-based definition rather than the previous similarity-style version.The note concerns results in Table 1 of the previous paper.
- Ablation study: The ablation study compares early concatenation, texture warping, feature warping, and Liquid Warping Block propagation under otherwise matched modules and losses.The four strategies are denoted WC, WT, WF, and WLWB.
4.2. Results of Human Appearance Transfer.
After training, the model can be applied directly to appearance transfer and preserves source identity while retaining clothing details from the reference, including for some out-of-domain reference images.
- Appearance transfer: Appearance transfer preserves the source face identity and clothing texture, color, and style while using clothing details from the reference image.The model is applied directly after training as part of the unified three-task system.
- Appearance transfer: The method produces decent appearance-transfer results even when reference images come from the Internet and are outside the iPER domain.The paper identifies the last five columns of Figure 8 as such examples.
4.3. Results of Human Novel View Synthesis.
For novel view synthesis, the method changes camera views across a wide angular range and predicts plausible invisible content while retaining source identity and clothing details under self-occlusion.
- Novel view synthesis: The method synthesizes views from 30° to 330° using randomly sampled source images from the iPER test set.The results are illustrated in Figure 9.
- Novel view synthesis: Under changed viewpoints, the method predicts reasonable invisible parts and preserves face identity and clothing details even with self-occlusion.The paper highlights the middle and bottom rows as self-occlusion cases.
5. Conclusion
The framework unifies human motion imitation, appearance transfer, and novel view synthesis using 3D body mesh recovery and Liquid Warping Block-based source-information propagation. It supports multiple-source warping and produces decent results while outperforming other methods.
- The framework handles human motion imitation, appearance transfer, and novel view synthesis in one unified model.
- Its body recovery module estimates a 3D body mesh, providing a more powerful representation than 2D pose.
- The Liquid Warping Block propagates source information in image and feature spaces while supporting flexible warping from multiple sources.
- Extensive experiments show that the framework outperforms other methods and produces decent results.