Source-linked AI summary
Synthesizing Images of Humans in Unseen Poses
Guha Balakrishnan, Amy Zhao, Adrian V. Dalca, Fredo Durand, John Guttag
TL;DR
The paper addresses synthesizing a person in an unseen target pose while retaining the person’s appearance and background. It uses a modular generative network that separates and moves body parts, fills background holes, and refines details with adversarial training. Across golf, yoga/workouts, and tennis, it reports accurate within- and cross-action synthesis and coherent videos from pose sequences.
Problem
The paper asks how to synthesize a person in a desired unseen pose while preserving the person’s appearance and background despite occlusions and disoccluded regions.
Method
A modular generative network separates body-part and background layers, moves and refines body parts, fills disoccluded background, and composites the result using paired images and poses.
Results
The method accurately reconstructs poses within action classes, transfers poses across action classes, and produces temporally coherent videos from pose sequences.
Takeaways & Limitations
Explicitly separating pose-related body layers from background synthesis supports cross-action pose-appearance combinations and coherent frame-by-frame video outputs.
Takeaways & Limitations
With only 2D joint locations, depth and multiple valid configurations near joints must be inferred; additional 3D joint information may be desirable as pose variety increases.
Abstract
from arXiv · showhide
We address the computational problem of novel human pose synthesis. Given an image of a person and a desired pose, we produce a depiction of that person in that pose, retaining the appearance of both the person and background. We present a modular generative neural network that synthesizes unseen poses using training pairs of images and poses taken from human action videos. Our network separates a scene into different body part and background layers, moves body parts to new locations and refines their appearances, and composites the new foreground with a hole-filled background. These subtasks, implemented with separate modules, are trained jointly using only a single target image as a supervised label. We use an adversarial discriminator to force our network to synthesize realistic details conditioned on pose. We demonstrate image synthesis results on three action classes: golf, yoga/workouts and tennis, and show that our method produces accurate results within action classes as well as across action classes. Given a sequence of desired poses, we also produce coherent videos of actions.
1. Introduction
The paper introduces automated synthesis of a person in a desired unseen pose while retaining the person’s appearance and background. It addresses pose changes through modular decomposition and demonstrates within-action, cross-action, and temporally coherent synthesis.
- The method synthesizes a person in a target pose from an input image while preserving the person’s appearance and background.
- Pose changes create moving parts, self-occlusions, shading and edge changes, and background regions that require plausible filling.
- A modular network separates body parts and background, moves parts, refines the foreground, fills disoccluded background, and composites the result using only a target image label.
- The layering strategy contrasts with latent-space transformation methods by decoupling foreground and background synthesis and explicitly modeling body-part movement.
- Experiments on golf, yoga/workouts, and tennis show accurate within-class reconstruction, cross-class pose transfer, and temporally coherent videos from pose sequences.
2. Related work
Prior view-synthesis methods commonly encode images and transformations into a latent space, whereas this work represents scenes as manipulatable layers. The approach also uses conditional adversarial training to improve realism in pose-conditioned image synthesis.
- Related human view-synthesis methods use encoder-decoder networks to capture relationships between an input image, a desired transformation, and the output image.
- This work instead represents scenes as separately manipulatable layers, enabling explicit body-part movement and background synthesis.
- Image-translation tasks modify pixels between representations, while this method incorporates a UNet architecture for several translation subtasks.
- A conditional GAN trains a generator to produce images conditioned on an input image and target pose while a discriminator distinguishes real from synthesized candidates.
- Figure 2 depicts the model input as source image, source pose, and target pose, with synthesized output trained against a target image.
3. Method
The method learns to translate pose changes into image space using a modular network trained on source images, source poses, target poses, and target images. It decomposes motion, occlusion, background completion, and detail synthesis into coordinated operations.
- Training uses tuples ((Is, ps, pt), It), where source image and poses condition synthesis of a target image depicting the same person, attire, and background.
- The network separates the scene into foreground, background, and body-part layers so arms and legs can be moved independently.
- Disoccluded background pixels receive synthesized appearance, while feature and adversarial losses promote realistic shading and high-frequency body details conditioned on target pose.
- The architecture is divided into four modular subtasks, beginning with source segmentation of body parts and background before spatially moving parts toward target locations.
D. Background Synthesis
The model synthesizes target backgrounds by separating body-part layers, transforming them to the desired pose, and filling pixels disoccluded by the foreground before compositing the result. The full network jointly learns these modular subtasks from the target image, while foreground synthesis refines body appearance and predicts the compositing mask.
- Network overview: Four jointly trained modules segment the source scene, move body parts, synthesize the foreground and mask, and fill the background before compositing.Modules A–D respectively perform segmentation, spatial transformation, foreground synthesis, and background hole-filling.
- Foreground spatial transformation: Body-part layers are transformed independently using pose-derived similarity transformations that account for translation, rotation and scale.Bilinear interpolation produces warped layers at approximately the target location, scale and orientation.
- Foreground synthesis: The foreground branch uses a UNet to merge transformed parts, refine appearance, and output both the target foreground and mask from shared decoding features.The target pose supplies spatial context for joint locations in the target image.
- Synthesis behavior: Foreground rendering remains realistic under dramatic pose changes, although pixels outside the predicted mask can be incoherent without affecting the loss.Held golf clubs and tennis rackets are not retained in some outputs because their movement patterns are inconsistent with the body pose.
- Background synthesis: The background branch receives background pixels plus Gaussian noise where the foreground was located, using pose context to synthesize hole-filled backgrounds.Noise initialization provides high-frequency gradients useful for texture synthesis.
- Background synthesis: Background synthesis handles varied colors and textures but sometimes leaves feet or held objects because the golf data contain static-foot bias and object motion is inconsistent.Examples include residual feet in a golf background and part of a tennis racket.
4. Experiments and Results
Experiments evaluate the method on YouTube action videos using within-action reconstruction, cross-action pose transfer, and pose-sequence video synthesis. VGG+GAN improves visual sharpness and realism, while the modular method outperforms a UNet baseline and produces realistic cross-action images and temporally coherent sequences.
- Evaluation Setup: The dataset combines 266 YouTube videos spanning golf, yoga/workouts, and tennis, with identity-disjoint video-level testing.Videos provide paired images and poses from the same video, while static backgrounds help hold appearance and background constant during pose changes.
- Evaluation Setup: The experiments test reconstruction from held-out target frames, coherent synthesis from pose sequences, and pose transfer across action classes.Cross-action examples include synthesizing a tennis player in a golfer’s pose.
- Within-Action Synthesis: Our method outperforms the UNet baseline on L1 error, VGG error, and SSIM, with statistically significant differences.Errors are lower-is-better, SSIM is higher-is-better, and the table reports standard deviations.
- Video Synthesis: Applying the method independently to pose sequences yields temporally coherent golf and tennis videos with consistent backgrounds and believable foreground textures.The synthesized textures do not exactly match the ground-truth frames, while backgrounds remain consistent because they use source image and pose information only.
- Loss Functions: VGG+GAN produces the sharpest images and the gradient distribution closest to real images, whereas L1 produces the blurriest outputs and most small gradients.VGG+GAN also captures subtle clothing texture, lighting, and body-detail effects.
- Cross-Action Synthesis: The model produces realistic images across all action-class permutations, including pose-appearance combinations absent from training.This result is reported despite training only on image pairs drawn from the same video.
5. Discussion
The model generalizes across action classes and produces temporally coherent pose sequences despite framewise application. Its main limitations are facial detail quality and ambiguity inherent in 2D pose-to-image synthesis.
- Generalization: Cross-action transfers show that the model generalizes to pose-appearance combinations unseen during training.The authors attribute this to disentangling some pose and appearance characteristics.
- Temporal consistency: Framewise application still produces temporally coherent appearances in synthesized action sequences.The method independently applies its model to each target pose without explicitly enforcing temporal consistency.
- Limitations: Facial detail remains a weakness, while L1 loss and SSIM do not distinguish subtle deformations such as malformed facial features.The authors identify better image-quality metrics as important for improving facial realism.
- Limitations: 2D pose-to-image synthesis is underconstrained because wrist locations do not determine depth or nearby body configurations.The authors suggest incorporating 3D joint locations as the number of possible poses increases.
6. Summary
The paper presents a modular layered network for synthesizing humans in new poses while preserving foreground appearance and background realism. Experiments show generalization across action classes and temporally coherent action videos.
- Method: The model decomposes scenes into background and body-part layers, moves parts to target locations, and decouples foreground from background.This layered design captures large pose changes while maintaining foreground appearance.
- Method: Layered synthesis produces more realistic backgrounds than a typical UNet architecture.The foreground and background are modeled separately.
- Results: The design generalizes beyond explicit training tasks by transferring poses across action classes and producing temporally coherent action videos.These videos are generated despite the network not being explicitly trained to preserve temporal continuity.