Source-linked AI summary
Shadow Art Revisited: A Differentiable Rendering Based Approach
Kaustubh Sadekar, Ashish Tiwari, Shanmuganathan Raman
TL;DR
The paper asks how to estimate an underlying 3D shape using only desired shadow images, without collecting or annotating 3D ground truth. The framework optimizes voxel- and mesh-based 3D representations through differentiable rendering using shadow images and their corresponding projection information. The authors report artistic effects beyond contemporary examples, including sculptures generated from half-toned face images and multi-view sketches.
Problem
The paper asks how to estimate an underlying 3D shape using only desired shadow images, without collecting or annotating 3D ground truth.
Source: §1 · p. 2Method
The framework optimizes voxel- and mesh-based 3D representations through differentiable rendering using shadow images and their corresponding projection information.
Source: §1 · p. 2, §3.1 · p. 3Results
The authors report artistic effects beyond contemporary examples, including sculptures generated from half-toned face images and multi-view sketches.
Source: §1 · p. 2Takeaways & Limitations
The framework is also applicable to reconstructing 3D shapes from sketches drawn from three viewpoints and to producing shadows of faces and animated movie characters.
Source: §6 · p. 8The voxel representation assumes the 3D object is enclosed in a known-size cube centered at the origin.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
While recent learning based methods have been observed to be superior for several vision-related applications, their potential in generating artistic effects has not been explored much. One such interesting application is Shadow Art - a unique form of sculptural art where 2D shadows cast by a 3D sculpture produce artistic effects. In this work, we revisit shadow art using differentiable rendering based optimization frameworks to obtain the 3D sculpture from a set of shadow (binary) images and their corresponding projection information. Specifically, we discuss shape optimization through voxel as well as mesh-based differentiable renderers. Our choice of using differentiable rendering for generating shadow art sculptures can be attributed to its ability to learn the underlying 3D geometry solely from image data, thus reducing the dependence on 3D ground truth. The qualitative and quantitative results demonstrate the potential of the proposed framework in generating complex 3D sculptures that go beyond those seen in contemporary art pieces using just a set of shadow images as input. Further, we demonstrate the generation of 3D sculptures to cast shadows of faces, animated movie characters, and applicability of the framework to sketch-based 3D reconstruction of underlying shapes.
1. Introduction
Shadow art uses carefully arranged 3D objects to create expressive shadows, but the paper describes little recent progress in this area. It introduces differentiable-rendering optimization to construct sculptures from shadow images and projection information, including artistic effects beyond contemporary examples.
1. Introduction
- The authors frame shadow art as a largely unexplored artistic application, despite examples by artists including Kumi Yamashita and Tim Noble and Sue Webster.They note that no significant progress has been observed since shadow art was introduced to the vision and graphics community.Source: §1 · pp. 1–2
- Differentiable-rendering methods use single- or multi-view 2D image supervision to estimate underlying 3D shape, eliminating 3D data collection and annotation required as accurate ground-truth supervision by most learning-based reconstruction methods.Source: §1 · p. 2
- The paper introduces differentiable-rendering optimization to create 3D sculptures that cast different shadows from input shadow images and corresponding projection information.It presents both voxel- and mesh-based pipelines.Source: §1 · p. 2
- The framework generates sculptures for artistic effects beyond contemporary examples, including from half-toned face images and sketches drawn from multiple viewpoints.The paper also describes generating different shadows when sculptures are lit from different directions.Source: §1 · p. 2
2. Related Work
Prior work explored shadow design and reconstruction, while differentiable-rendering methods vary by 3D representation. This paper focuses on voxel and mesh pipelines and contrasts its approach with earlier shadow-art optimization.
Shadow Art
- Earlier shadow research includes an interface for constrained cinematic shadow design and reconstruction approaches based on visual hulls and silhouette-constrained min-cuts.The visual hull is described as the intersection of generalized cones from silhouettes and camera locations.Source: §2 · pp. 2–3
Relation with the state-of-the-art method
- The closest prior shadow-art method, Mitra et al., deforms input images to handle inconsistencies, whereas this work uses arbitrary shadow images without explicit image deformation.The paper notes its resulting objects need not exist in the real world, though they cast physically realizable shadows under appropriate lighting.Source: §2 · p. 3
Differentiable Rendering
- Differentiable-rendering methods use point clouds, voxels, meshes, or neural implicit representations; this work focuses on voxel and mesh representations.The authors describe both representations' strengths and weaknesses for creating plausible shadow art.Source: §2 · p. 3
- For its mesh pipeline, the paper deploys Soft Rasterizer [5], which replaces rasterization with a soft version to make it differentiable.The cited work uses a deformable template mesh for training and reports reconstruction results.Source: §2 · p. 3
3. Method
The method optimizes a 3D sculpture to cast a set of specified shadows, treating shadow views as silhouettes while allowing the resulting sculpture to be abstract. It describes voxel- and mesh-based differentiable-rendering pipelines with image-based and mesh-regularization losses.
3.1. Problem Formulation
- Shadow art seeks one 3D sculpture that casts the specified silhouettes, which may be arbitrary views and need not depict the same real-world object.The input pairs each target shadow image with projection information; multiple sculptures may produce the same shadows.Source: §3.1 · p. 3
3.2. System Overview
- The system overview identifies target shadow images and their associated projections as the inputs and presents the mesh-based pipeline's information flow.The target images and projections are denoted I and P, and rendered images from the learned sculpture are denoted eI.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Voxel Based Optimization
- The voxel pipeline represents geometry with learnable density values in a bounded 3D grid and optimizes them against target shadows using a weighted image loss.Voxel rendering aggregates contributions along rays; the resulting voxel representation can be converted to a mesh by thresholding density values.Source: §3.3 · p. 4
3.4. Mesh Based Optimization
- The mesh pipeline initializes a source mesh and learns per-vertex displacements so its rendered silhouettes match the desired shadows.The mesh is represented by vertices connected into triangular faces and is rendered from the associated projections.Source: §3.4 · p. 5
3.4.1 Loss Function
- Mesh optimization combines image loss with normal-consistency, Laplacian, and edge-length regularization to smooth the sculpture and constrain deformations.Normal consistency encourages smoothness, Laplacian smoothing limits large deformations, and edge-length regularization prevents flying vertices.Source: §3.4.1 · p. 5
3.5. Implementation Details
- The implementation uses PyTorch3D, a level-4 icosphere mesh initialization, and a 128×128×128 voxel grid inside a cube of side length 1.7 world units.The passage also reports 128 × 128 custom silhouettes and training settings for the optimization pipelines.Source: §3.5 · p. 5
4. Experimental Analysis
The experiments assess shadow agreement and mesh quality, examine regularization, and compare voxel and mesh pipelines with prior work across varied view settings.
4.1. Evaluation Metrics
- The evaluation uses IoU and Dice Score to compare rendered and target silhouettes, and normal consistency as a no-reference measure of mesh quality.The authors use a no-reference measure because they lack 3D ground truth for the optimized shape.Source: §4.1 · p. 5
4.2. Ablation Study
- Image-based loss alone leaves spike-like artifacts, while edge-length regularization reduces spikes, Laplacian smoothing limits large deformations, and normal consistency further smooths the surface.The combined objective yields the mesh shown in Figure 5(d).Source: §4.2 · p. 6, Figure 5 · p. 6
4.3. Qualitative and Quantitative Analysis
- Across orthogonal and non-orthogonal views, voxel and mesh renderings have comparable IoU and Dice scores, while mesh sculptures have higher normal consistency.Voxel shapes are less smooth; their normal consistency is measured after converting them to meshes.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
- Mesh rendering can leave discontinuities near nonconvex regions, while voxel rendering can produce outlier voxels and discretized shapes with high memory and computational requirements.The mesh limitation is attributed to the icosphere's difficulty with sharp discontinuities, especially under regularization; voxel methods handle sharp discontinuities and holes more effectively.Source: §4.3 · p. 7
- Both differentiable-rendering pipelines produce plausible sculptures and outperform by a large extent in handling shadow inconsistencies without deforming target shadows.Figure 4 presents qualitative and quantitative results across two-view, three-orthogonal-view, and three-non-orthogonal-view settings.Source: Figure 4 · p. 6, §4.3 · p. 7
4.4. Comparison with the State-of-the-art method
- Compared with, the voxel pipeline renders shadows more consistent with targets without deforming the input images; introduces such deformation to seek consistency.Figure 6 highlights inconsistencies in orange.Source: §4.4 · pp. 7–8, Figure 6 · p. 7
5. Applications
The framework generates shadow-art sculptures depicting scientists and movie characters, and also extends to half-toned shadow images and sketch-based 3D modeling.
5. Applications
- The approach generates faces of well-known scientists and movie characters such as Minions and Ironman as shadow-art sculptures.The examples demonstrate its use with complex objects or scenes.Source: §5 · p. 8
- Half-toned images, in addition to binary silhouettes, can be used to generate 3D shadow-art sculptures.Figure 1 illustrates this use with face sketches.Source: §5 · p. 8
- Hand-drawn sketches from three viewpoints are used to reconstruct 3D objects, including a flower vase, pen-stand, and coffee mug.The examples are shown in Figures 8 and 1.Source: §5 · p. 8
6. Conclusion
The paper presents voxel- and mesh-based differentiable-rendering pipelines for generating 3D shadow-art sculptures from shadow images and projection information. It also demonstrates sketch-based 3D reconstruction and identifies dynamic shadow art as future work.
6. Conclusion
- The framework generates 3D shadow-art sculptures from shadow images and their associated projection information using voxel- and mesh-based rendering pipelines.The paper identifies visually plausible, consistent shadows of rigid objects, faces, and animated movie characters as its focus.Source: §6 · p. 8
- The framework is also demonstrated for reconstructing 3D shapes from sketches drawn from three viewpoints.Source: §6 · p. 8
- The work focuses on static sculptures and static shadows; dynamic shadow art is proposed as a future direction.Source: §6 · p. 8