PaperlayerView exact revision on arXiv

Source-linked AI summary

Hand-Shadow Poser

Hao Xu, Yinqiao Wang, Niloy J. Mitra, Shuaicheng Liu, Pheng-Ann Heng, Chi-Wing Fu

arXiv:2505.07012v1cs.CGcs.AI

TL;DR

Hand-Shadow Poser addresses the challenge of recovering two hand poses from a target shadow, where many poses can explain an ambiguous, colorless mask. It uses a three-stage pipeline to generate hand assignments, align coarse poses, and refine shadow features; evaluations report robust results across varied shapes, with several stated scope limits.

  • Problem

    Recovering two hand poses from a target shadow is ambiguous because many 3D poses can produce similar shadows, while colorless masks provide limited shape information.

    Source: §1 · p. 2, §5 · p. 6

  • Method

    Hand-Shadow Poser generates diverse left-right hand assignments, aligns them to obtain coarse poses, and refines poses for shadow features and physical plausibility.

    Source: §1 · p. 2, §6 · p. 8, §8 · p. 14

  • Results

    The evaluation reports generalizability and robustness across varied input shapes, with prominent shadow features preserved.

    Source: §8 · p. 14

  • Takeaways & Limitations

    The resulting hand poses can be reproduced by human hands and 3D printing.

    Source: §8 · p. 14

    The method may not reproduce overly complicated shadows with small, thin structures and assumes a fixed light source.

    Source: §8 · p. 14

Abstract

from arXiv · show

Hand shadow art is a captivating art form, creatively using hand shadows to reproduce expressive shapes on the wall. In this work, we study an inverse problem: given a target shape, find the poses of left and right hands that together best produce a shadow resembling the input. This problem is nontrivial, since the design space of 3D hand poses is huge while being restrictive due to anatomical constraints. Also, we need to attend to the input's shape and crucial features, though the input is colorless and textureless. To meet these challenges, we design Hand-Shadow Poser, a three-stage pipeline, to decouple the anatomical constraints (by hand) and semantic constraints (by shadow shape): (i) a generative hand assignment module to explore diverse but reasonable left/right-hand shape hypotheses; (ii) a generalized hand-shadow alignment module to infer coarse hand poses with a similarity-driven strategy for selecting hypotheses; and (iii) a shadow-feature-aware refinement module to optimize the hand poses for physical plausibility and shadow feature preservation. Further, we design our pipeline to be trainable on generic public hand data, thus avoiding the need for any specialized training dataset. For method validation, we build a benchmark of 210 diverse shadow shapes of varying complexity and a comprehensive set of metrics, including a novel DINOv2-based evaluation metric. Through extensive comparisons with multiple baselines and user studies, our approach is demonstrated to effectively generate bimanual hand poses for a large variety of hand shapes for over 85% of the benchmark cases.

1 INTRODUCTION

Hand-Shadow Poser tackles the inverse problem of finding two hand poses that reproduce a target shadow, despite pose ambiguity and limited shadow information. Its three-stage pipeline separates hand anatomy from shadow semantics, and the authors report effectiveness across diverse shapes and evaluations.

1 INTRODUCTION

  • Hand-shadow art uses shadows cast by hands to reveal shapes of objects and has a long history across cultures.Its appeal is described in terms of simplicity, flexibility, and creativity.Source: §1 · p. 1
  • The task is to infer left- and right-hand 3D poses whose combined shadow resembles a given target shape.The paper presents this as an inverse problem and demonstrates shapes ranging from animals to alphanumeric characters and intricate forms.Source: §1 · p. 2
  • Recovering bimanual poses is ambiguous because many 3D poses can cast similar shadows, while colorless, textureless inputs make shape and feature recovery ill-posed.Hand anatomy also restricts fine-grained preservation of shadow features, and specialized shadow training data are scarce.Source: §1 · p. 2
  • The authors introduce a three-stage pipeline: generate left-right hand-shape hypotheses, align them to recover coarse poses, then refine poses for salient shadow features and physical plausibility.The feed-forward stages are trained on generic public hand datasets with augmentation, avoiding extensive specialized hand-shadow training data.Source: §1 · p. 2
  • Quantitative comparisons, qualitative results, and user studies consistently show the approach’s effectiveness and robustness.The evaluation includes diverse 2D masks and metrics focused on perception, semantics, and salient characteristics.Source: §1 · pp. 2–3
  • The evaluation benchmark contains 210 varied shadow art forms, alongside shadow-specific metrics for quality assessment.The benchmark includes diverse masks, and the metrics focus on perception, semantics, and salient characteristics.Source: §1 · pp. 2–3

2 RELATED WORK

The related work spans computational visual art, optical illusions, shadow art, and hand-pose recovery from silhouettes. The paper positions its task among prior shadow-generation and pose-estimation methods while distinguishing its bimanual, shadow-specific approach.

Computational visual art and optical illusions

  • Computational visual-art research spans paintings, 3D scenes, sculptures, and reliefs, while optical-illusion work includes 2D image effects and 3D shapes designed to appear differently under projection or viewing changes.The paper characterizes its task as lying at the intersection of 2D and 3D optical illusions.Source: §2 · p. 3

Shadow art

  • Prior shadow-art work includes transforming or manipulating shadows, constructing objects with multiple projected patterns, and generating target shadows with objects or differentiable rendering.Other approaches use implicit representations, jointly optimize lighting and screen orientation, or create shadow art from real items.Source: §2 · p. 3

Hand-shadow creation

  • Earlier silhouette-to-body methods require professional contour-to-body hints or can be sensitive to initialization and local optima for this task.A related hand-shadow method directly minimizes image loss with differentiable rendering, but showcases only a few examples and has similar optimization issues.Source: §2 · p. 4
  • The paper contrasts its approach with prior work by claiming broader hand-shadow coverage, salient-characteristic capture, and generative proposals of diverse anatomically constrained poses.It also reports that differentiable rendering alone does not achieve its results.Source: §2 · p. 4

3D hand pose estimation from silhouettes

  • Silhouette-based 3D hand-pose methods estimate single-hand poses, but bimanual masks obscure the separate hand shapes and require jointly locating and posing both hands [Lee et al. 2019; Chang et al. 2023].The cited single-hand silhouette approach requires depth supervision during training [Lee et al. 2019].Source: §2 · p. 4

3 OVERVIEW

The task maps a binary target-shadow mask to 3D poses for two hands under a fixed light-and-screen setup. The overview motivates a three-stage approach with diverse hand assignments, coarse pose recovery, and feature-aware refinement.

Problem definition

  • The input is a binary target-shadow mask, and the output is a pair of MANO 3D hand poses whose projected shadow should match the target.The hands sit between a light source and screen plane.Source: §3 · p. 4
  • The setup fixes the light source and screen, aligns them horizontally and vertically with the hands, and excludes other body parts and objects.A small, intense light source and flat projection screen are used for clear, sharp shadow boundaries.Source: §3 · p. 4

Challenges

  • Direct shadow-difference optimization is initialization-sensitive and may reach local optima; pixel-perfect matching is infeasible under hand-anatomy limits, so distinctive features take priority.Manually specifying contour hints is described as impractical.Source: §3 · pp. 4–5
  • Feed-forward prediction also faces scarce annotated hand-shadow data and ambiguity: one target may admit many different two-hand poses.The method must produce diverse yet reasonable results and generalize beyond a specific data domain.Source: §3 · p. 5

Overview of our Hand-Shadow Poser

  • The overview divides the pipeline into diverse left-right mask assignment, coarse per-hand pose recovery with candidate selection, and iterative refinement for shadow resemblance and physical plausibility.The stages are intended to provide evaluations of output quality and the proposed designs’ effectiveness.Source: §3 · p. 5

4 GENERATIVE HAND ASSIGNMENT

Hand assignment predicts plausible 2D left- and right-hand masks for a target shadow, without requiring 3D poses at this stage. A conditional generative model explores multiple assignments, trained with augmented public hand data.

4 GENERATIVE HAND ASSIGNMENT

  • Hand assignment finds rough 2D hand masks with reasonable anatomy to match the input shadow, assigning each hand to different parts without requiring 3D hand poses at this stage.The input is only a binary mask, and many different hand shapes may match and form the target shadow.Source: §4 · p. 5
  • The authors report that deterministic segmentation misses assignment uncertainty, overemphasizes pixel-level accuracy, and performs worse for this textureless input.The lack of color and texture makes learning harder than in conventional segmentation tasks.Source: §4 · p. 5
  • A conditional denoising diffusion model generates diverse hand assignments from the input shadow mask, using classifier-free guidance for shadow-controlled multi-hypothesis generation.The model progressively denoises the intermediate output toward a two-channel assignment.Source: §4 · p. 5
  • Random noise seeds produce diverse assignments at inference, and swapping left and right hands adds mirror-symmetric alternatives.DDIM samples at arbitrary timestamps to speed inference.Source: §4 · p. 5, Figure 5 · p. 5
  • Training uses separate L2 losses for the predicted left- and right-hand masks against their ground-truth masks.The loss is weighted by the total noise variance at the diffusion step.Source: §4 · pp. 5–6
  • Generic public hand datasets supply training data, augmented by splicing left- and right-hand samples and rendering interacting hands from multiple views.Splicing synthesizes more diverse two-hand interactions, while multi-view rendering enriches viewing-angle diversity.Source: §4 · p. 6, Figure 6 · p. 6

5 GENERALIZED HAND-SHADOW ALIGNMENT

The alignment stage maps rough single-hand masks to coarse 3D poses, using generalized hand priors and similarity-based ranking to select hypotheses for refinement.

5 GENERALIZED HAND-SHADOW ALIGNMENT

  • The network recovers a coarse 3D pose for each single-hand mask; fine-grained shadow features are deferred to the refinement stage.Each pose comprises MANO coefficients for hand orientation, joint poses, shape, and wrist position.Source: §5 · p. 6
  • Colorless, sparse masks and rough hand-shape estimates make 3D pose recovery uncertain, motivating a model robust to input ambiguity.The inputs provide limited information, and shadow ambiguity means the mask only approximates the actual hand shape.Source: §5 · p. 6
  • A transformer initialized from RGB-pretrained weights is fine-tuned on large collections of real and synthetic binary hand masks to handle rough inputs.The training data include single and interacting hands.Source: §5 · p. 6
  • Unlabeled images receive MANO pseudo-labels from an RGB-pretrained network, then are rendered into paired masks and poses alongside labeled samples.This semi-supervised strategy avoids requiring highly accurate pseudo-labels for the original images.Source: Figure 7 · p. 6, §5 · p. 7
  • At inference, rendered pose hypotheses are scored against the target with LPIPS and DINOv2, then the top K are selected for refinement.The render-and-compare procedure ranks hypotheses despite the absence of inference-time ground truth.Source: §5 · p. 7, Figure 8 · p. 7

6 SHADOW-FEATURE-AWARE REFINEMENT

The refinement stage iteratively adjusts coarse hand poses to preserve salient shadow features while enforcing physical plausibility through anatomical, penetration, and distance constraints.

6 SHADOW-FEATURE-AWARE REFINEMENT

  • The final stage refines coarse poses to match both the shadow’s global shape and salient features while respecting anatomical constraints.DINOv2-based saliency guidance highlights characteristic shadow features during iterative optimization.Source: §6 · p. 7, Figure 10 · p. 7
  • Differentiable rendering optimizes joint angles and wrist positions while holding hand shape parameters fixed.The coarse poses from earlier stages provide an initial condition associated with faster and better convergence.Source: §6 · p. 7
  • DINOv2 saliency weights prominent shadow regions instead of treating every pixel equally, avoiding manually specified hint points.The figure illustrates target shadows above saliency maps with characteristic features highlighted in red.Source: §6 · p. 7, Figure 9 · p. 7
  • Anatomical plausibility is encouraged by penalizing abnormal joint-axis components using the twist-splay-bend frame.Source: §6 · p. 8
  • Penetration penalties cover both inter-hand intrusion and self-penetration.Inter-hand loss uses distances from penetrating vertices to the closest vertices on the other hand; self-penetration uses conic distance fields.Source: §6 · p. 8
  • A wrist-distance constraint limits excessive separation, since projection-space optimization can leave the hands too far apart in depth.The loss penalizes wrist separation only when it exceeds a threshold.Source: §6 · p. 8
  • The optimization combines similarity, anatomy, penetration, and distance losses in a weighted objective.Adam performs gradient-based optimization for a specified number of iterations.Source: §6 · p. 8

7 RESULTS AND EXPERIMENTS

The evaluation spans a 210-mask benchmark, quantitative and perceptual comparisons, real-hand demonstrations, and 3D-printed poses. Results support broad shape coverage, while ablations and failure analyses identify both useful components and boundaries.

7.1 Experimental Setup

  • The benchmark contains 210 masks: 62 alphanumeric characters, 87 real hand-shadow-art shapes, and 61 everyday-object shapes.Evaluation includes LPIPS, CLIP-Global, CLIP-Semantic, DINO-Global, and DINO-Semantic metrics.Source: §7.1 · pp. 8–9

7.2 Evaluation

  • Hand-Shadow Poser achieves the best results across all seven metrics and three benchmark classes in the quantitative comparison.The metrics assess visual similarity and human-rated global shape and semantic detail.Source: Table 1 · p. 11, §7.2 · pp. 11–12
  • Generated poses preserve distinctive details in examples and produce shadows across characters, classical shadow art, and everyday objects.Real-hand demonstrations and 3D prints further show results produced beyond rendered outputs.Source: §7.2 · p. 11
  • The method’s optimization runtime is around one-sixth of Baseline 1’s and half of Baseline 3’s.The authors attribute faster convergence alongside better-optimized shadows to the coarse poses from the first two stages.Source: §7.2 · p. 12, Table 2 · p. 12

7.3 User Studies

  • Across three shape classes, users rated the method above all baselines for both global shape similarity and local-detail preservation.Participants scored outputs on 1–5 Likert scales for Human-Global and Human-Semantic.Source: §7.3 · p. 12
  • With generated 3D hand models as references, participants reduced reproduction time from 121.4 to 65.8 seconds and improved average quality from 2.4 to 4.0.The demonstration study involved six target shadows reproduced by participants using their hands.Source: §7.3 · pp. 12–13

7.4 Model Analysis

  • Removing refinement significantly lowers DINO-Semantic scores, while removing saliency guidance degrades DINO-Global and CLIP-Semantic and weakens prominent-detail preservation.The authors also report physical artifacts when anatomy, penetration, or hand-to-hand distance constraints are removed.Source: Table 3 · p. 12, §7.4 · p. 13
  • Four distinct, physically feasible solutions can closely match one input, and coarse poses remain approximately consistent with assigned shapes under varying uncertainty.The method may struggle with arbitrary inputs, especially intricate details or thin structures.Source: §7.4 · pp. 13–14, Figure 20 · p. 14

8 CONCLUSION, LIMITATIONS, AND FUTURE WORKS

The paper presents Hand-Shadow Poser for creating hand-shadow art from 2D shapes and reports broad performance across comparisons and user studies. It also identifies constraints involving shape complexity, individual hand geometry, lighting, body feasibility, and forearms.

8 CONCLUSION, LIMITATIONS, AND FUTURE WORKS

  • Hand-Shadow Poser inversely creates hand-shadow art from 2D inputs using a three-stage pipeline that separates hand anatomy constraints from shadow-shape semantics.The framework uses generative hand assignment, generalized hand-shadow alignment, and shadow-feature-aware refinement.Source: §8 · p. 14
  • The authors report superior performance in quantitative and qualitative comparisons with baselines and in user studies, with evaluation across diverse shapes and prominent-feature preservation.They also report that results can be reproduced by human hands and 3D printing.Source: §8 · p. 14
  • The method may not reproduce overly complicated shapes with small or thin structures, and applying it to a particular person requires specifying that individual’s hand-shape parameters.Other stated limitations include a fixed-light-source assumption, infeasible two-hand poses, and omission of the forearm.Source: §8 · p. 14
  • Future directions include animated shadow art, hand-held objects, and coordinating hands from multiple artists.Source: §8 · p. 14
Loading 2505.07012v1…