Source-linked AI summary
Photo Wake-Up: 3D Character Animation from a Single Photo
Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman
TL;DR
The paper addresses how to animate a clothed human from a single photo as a fully 3D experience. It fits and warps a posable template to the photographed silhouette, handles selected self-occlusions, and produces a rigged mesh for animation. Across 70 examples, human evaluations rated the animations 2.76/3 on average, and 86% preferred the method over a compared prior result.
Problem
Animating a clothed person in fully 3D from one photo requires matching complex silhouettes and partial self-occlusions that initial animatable body fits do not capture.
Method
The method fits a SMPL template, aligns its silhouette through 2D warping, lifts warped normal and skinning maps into a rigged mesh, and uses label maps for selected self-occlusions.
Results
The method produces convincing animations across 70 whole-body, fairly frontal photos, with an average human-study score of 2.76/3 and 86% preference over a compared prior method.
Takeaways & Limitations
The system enables interactive 3D re-posing and augmented-reality viewing of photographed people emerging from photos.
Takeaways & Limitations
The method does not guarantee metrically correct geometry and remains limited to particular occlusions, while back appearance often requires simple inpainting or user interaction.
Abstract
from arXiv · showhide
We present a method and application for animating a human subject from a single photo. E.g., the character can walk out, run, sit, or jump in 3D. The key contributions of this paper are: 1) an application of viewing and animating humans in single photos in 3D, 2) a novel 2D warping method to deform a posable template body model to fit the person's complex silhouette to create an animatable mesh, and 3) a method for handling partial self occlusions. We compare to state-of-the-art related methods and evaluate results with human studies. Further, we present an interactive interface that allows re-posing the person in 3D, and an augmented reality setup where the animated 3D person can emerge from the photo into the real world. We demonstrate the method on photos, posters, and art.
1. Introduction
The paper “wakes up” a single photo by turning its foreground character into an animatable 3D subject that can emerge into augmented or virtual reality. It addresses silhouette mismatch and partial self-occlusion through 2D warping rather than directly deforming the 3D mesh.
- Motivation and contribution: The system brings a foreground character from a single photo to life as an interactive 3D animation and AR or VR experience.Unlike cinemagraphs, it creates a fully 3D experience from one photo rather than a 2D motion effect from video.
- Motivation and contribution: The central reconstruction challenge is matching the clothed subject’s complex silhouette, including self-occluded body parts, while preserving animation.Initial segmentation, pose estimation, and morphable-model fitting produce an animatable result that does not conform naturally to the photographed silhouette.
- Technical contribution: The method maps silhouette alignment into 2D, warps projected model maps, and lifts the result back into 3D instead of deforming the mesh directly.This avoids difficult mesh deformation around intricate regions such as fingers and supports complex silhouettes.
- Technical contribution: Body-part label maps extend the method to certain partial self-occlusions, particularly when an arm crosses the torso or other body regions.The supported operating range is limited to specific partial occlusions rather than all self-occlusion configurations.
- Scope and limitations: The system targets whole-body, fairly frontal photos and does not guarantee metrically correct 3D geometry because single-image reconstruction is inherently ambiguous.Automatic detection, segmentation, and skeleton fitting can remain unreliable, especially for abstract artwork, and back appearance must be hallucinated or user-guided.
- Novelty: The authors present the system as, to their knowledge, the first to animate a clothed subject in 3D from a single image.They distinguish it from prior work using primarily 2D or pseudo-3D results or requiring video input.
2. Related Work
Prior work includes video-based, multi-view, image-collection, and primarily 2D or pseudo-3D animation approaches. This paper positions itself as single-photo, clothed-subject, fully 3D animation with an animatable reconstruction.
- Animation from video and collections: Earlier animation methods use videos, multi-view captures, or image collections to create human, clothing, facial, animal, or scene transformations.Examples include infinite-length video textures, motion graphs, multi-view human animation, and temporal changes assembled from photo datasets.
- Animation from single images: Single-photo methods have produced effects such as region motion, animal motion cycles, object viewpoint changes, and video-guided facial animation.These works demonstrate compelling animation from still imagery but address varied objects, animals, or facial motion rather than the paper’s target setting.
- Single-image body reconstruction: Single-image body reconstruction research estimates SMPL pose and shape, volumetric bodies, or dense texture correspondences, while multi-view work uses additional views.These methods provide relevant reconstruction foundations but do not necessarily yield the paper’s clothed, animation-ready solution.
- Positioning: Most single-image person animation remains primarily 2D or pseudo-3D, whereas this paper aims for a fully 3D experience from a clothed subject.The closest 3D reconstruction work cited uses video input, and the paper compares against both categories in its experiments.
3. Overview
The pipeline extracts a person’s image evidence, fits a SMPL template, and converts silhouette-aligned 2D model maps into a complete rigged mesh. The resulting mesh can then be animated with arbitrary 3D motion sequences.
- Input analysis: The system first applies person detection, segmentation, and 2D pose estimation to the input photo.The implementation uses off-the-shelf algorithms, with segmentation refined before the former person region is inpainted.
- Template fitting: A SMPL template is fit to the estimated 2D pose and projected into the image as normal and skinning maps.These maps provide the geometric and rigging information that will later be warped and lifted into 3D.
- Mesh recovery: The core process aligns the SMPL silhouette to the person silhouette, warps the normal and skinning maps, integrates depth, and repeats the process for an imputed back view.Combining front and back depth and skinning information produces a complete rigged 3D mesh.
- Animation: After mesh construction, the reconstructed person is textured and animated with motion-capture sequences over an inpainted background.The overall rigged mesh can also accept any 3D motion sequence.
4. Mesh Construction and Rigging
The paper recovers an animatable textured mesh by warping a fitted SMPL representation in image space to the clothed silhouette, then lifting the warped maps into 3D. This avoids difficult direct mesh deformation and supports selected arm-over-body occlusions.
- Mesh construction: The key technical goal is recovering an animatable, textured 3D mesh from one photo that fits the photographed subject.The fitted SMPL model is a useful starting point but is semi-nude and fails to match the person’s clothed silhouette.
- Mesh construction: Directly optimizing SMPL mesh vertices to match silhouettes is difficult around fingers because it must avoid boundary violations, pinching, and self-intersection.This difficulty motivates the paper’s image-space alternative.
- 2D warping: The proposed approach warps the SMPL silhouette and its projected normal and skinning maps in 2D, then lifts front and imputed back views into a rigged 3D mesh.The recovered mesh agrees with the input silhouettes and is ready for animation alongside the fitted 3D skeleton.
- Self-occlusion: The method separately addresses arm-over-body self-occlusion as a dedicated extension of the mesh-construction pipeline.The section treats self-occlusion after introducing the 2D warping construction.
4.1. Mesh Warping, Rigging, & Skinning
The method warps a fitted SMPL model in 2D to match the person’s silhouette, then lifts warped maps into a rigged 3D mesh. It uses mean-value-coordinate correspondence, normal integration, hole filling, and front/back reconstruction.
- 4.1. Mesh Warping, Rigging, & Skinning: The pipeline fits SMPL to the 2D pose, projects depth, normal, and skinning maps, and warps them from the SMPL silhouette to the photo silhouette.The warped maps are used to construct output depth, normal, and skinning maps for rigging.
- 4.1. Mesh Warping, Rigging, & Skinning: Warping normals and integrating them avoids the excessive z-direction flattening produced by directly warping SMPL depth maps.The direct-depth alternative is especially flat around the head, while normal integration produces the final depth map.
- 4.1. Mesh Warping, Rigging, & Skinning: Mean-value coordinates define a smooth inverse warp over the silhouette, while dynamic programming matches boundary points with proximity and monotonicity constraints.The correspondence function links input-silhouette boundary points to SMPL boundary points before constructing the warp.
- 4.1. Mesh Warping, Rigging, & Skinning: Small regions outside the SMPL silhouette can appear after warping, so the method smoothly fills holes in the warped normal and skinning maps.These holes arise from non-bijective correspondence between the input and SMPL silhouette boundaries.
- 4.1. Mesh Warping, Rigging, & Skinning: The method mirrors and warps a rendered SMPL back view, reconstructs front and back meshes separately, and stitches them at their corresponding boundaries.The resulting complete mesh carries the corresponding skinning weights for animation.
4.2. Self-occlusion
Self-occlusion is handled by segmenting the body into labeled parts, identifying occlusion boundaries, completing partially hidden shapes, and reconstructing each part before assembly. The demonstrated operating range focuses on partial arm-over-body occlusions with connected covered regions.
- 4.2. Self-occlusion: The method segments the body into parts, completes partially occluded segments, and reconstructs each part before assembling the final mesh.A single depth map from a binary silhouette is insufficient when one body part covers another.
- 4.2. Self-occlusion: The supported case is partial arm crossing over the head, torso, or legs when each covered region remains a single connected component; other cases remain unhandled.The authors identify crossed legs during sitting as an example of an unsupported occlusion scenario.
- 4.2.1 Body label map: The initial label map is optimized to remain similar to the projected SMPL labels while encouraging spatially coherent body-part assignments.The initial map is obtained with an MRF objective and approximate α-expansion optimization.
- 4.2.1 Body label map: The refinement stage detects label discontinuities that are not neighbors on the 3D body, then uses warped SMPL boundaries to remove spurious occlusion boundaries.The refined map is additionally fit to image color discontinuities within the occlusion mask.
- 4.2. Self-occlusion: After recovering unoccluded and occluded shapes, the method projects SMPL depth, normals, and skinning weights part-by-part, builds part meshes, and assembles them with boundary smoothing.Laplacian smoothing reduces jagged artifacts caused by binary silhouette segmentation.
5. Final Steps
The final pipeline corrects head pose using facial fiducials and provides front projection plus back-texture completion options. It illustrates the resulting label maps and meshes, while noting that detailed texturing procedures are supplemental.
- 5. Final Steps: Head pose is corrected by aligning projected 3D facial fiducials with detected image fiducials, followed by a smooth warp of the reconstructed head.The procedure is used because the fitted SMPL head pose is often inaccurate.
- 5. Final Steps: The front texture is projected from the image, occluded frontal regions are filled with PatchMatch, and the back is mirrored or inpainted with optional user guidance.Back inpainting uses body label maps to draw texture from regions with matching labels.
- 5. Final Steps: Figure 6 shows computed body label maps and reconstructed meshes for example input photos.The input photos appear in the upper-right corner of the examples.
6. Results and Discussion
The method produces animatable 3D people from single photos, with interactive pose editing and augmented-reality presentation. Across examples and comparisons, the reconstructed meshes support convincing animation while outperforming related approaches, though several limitations remain.
- Applications and interface: The system was tested on 70 Internet photos spanning art, posters, and graffiti, producing animation and augmented-reality results across varied environments.The interface supports viewpoint changes and pose editing, while AR allows users to place artwork on a wall and walk around the playing animation.
- Applications and interface: Users can modify animations, drag joints to edit poses, and adjust detection, segmentation, body labels, and mirrored textures through the interface.When needed, interaction for correcting detection, segmentation, labels, or textures takes seconds.
- Comparison with prior work: 86% of 103 participants preferred the proposed animation over as more realistic.The comparison used ’s only example with substantial out-of-plane motion, and the proposed method also enables 3D experiences unavailable in.
- Comparison with prior work: Compared with fitted SMPL and the method of [1], the proposed mesh better matches complex silhouettes and preserves detailed fingers and self-occlusion handling.The fitted SMPL model does not correctly handle subject silhouettes, while [1] fails on complex regions such as hair and fingers and does not handle self-occlusion well.
- Limitations: The method does not model shadows or reflections, may produce incorrect poses or unrealistic shapes, and does not cover all self-occlusion cases.Its supported self-occlusions are limited to arms partially occluding the head, torso, or legs; crossed legs while sitting remain future work.