Source-linked AI summary
Text-Driven Artistic Staging: Pose, Lighting, and Camera References from Paintings
Yunge Wen
TL;DR
Existing methods generally separate pose, lighting, and camera, despite their joint role in emotional staging. The paper reconstructs editable 3D staging from paintings, trains a text-conditioned flow-matching transformer, and achieves 32.2% retrieval R@1 versus 16.6% for CLIP retrieval while approximately matching corpus-level diversity. The authors identify limited pose coverage, assumed field of view, close multi-figure placement, and approximate lighting as scope boundaries.
Problem
Existing generative methods model pose, lighting, and camera independently rather than jointly generating editable 3D staging from affective text.
Method
The paper reconstructs 11,911 text–staging pairs from 2,328 paintings and trains a flow-matching transformer to jointly generate variable-figure pose, lighting, and camera tokens from text.
Results
32.2% R@1 versus 16.6% for retrieval, with 0.589 multimodality and approximately matched corpus-level diversity at guidance weight w=6.
Takeaways & Limitations
The results demonstrate the feasibility of generating editable, emotionally conditioned 3D staging references from text.
Takeaways & Limitations
The corpus is dominated by portraits, while field of view is assumed, multiple figures may be placed closer than in source paintings, and lighting is only an approximate low-frequency cue.
Abstract
from arXiv · showhide
Artists coordinate human pose, illumination, and camera placement to convey narrative and emotion, but existing generative methods typically model these elements independently. We introduce text-to-editable 3D staging, a task that jointly generates human poses, a dominant light, and a camera configuration from an affective description. We construct 11,911 text--staging pairs from 2,328 figurative paintings by reconstructing SMPL bodies, estimating low-frequency illumination, recovering camera parameters, and pairing each scene with ArtEmis descriptions. We train a flow-matching transformer that supports variable numbers of figures and produces multiple staging alternatives for each prompt. On held-out descriptions, the model achieves 32.2\% retrieval R@1, compared with 16.6\% for CLIP-based nearest-neighbor retrieval, while approximately preserving corpus-level diversity. These results demonstrate the feasibility of generating editable, emotionally conditioned 3D staging references from text.
1 Introduction
The paper treats pose, lighting, and camera placement as a coupled staging problem for conveying emotion and narrative. It reconstructs this joint solution from paintings and generates editable 3D staging alternatives from affective text.
- Motivation: Composition jointly coordinates body pose, light placement, and viewer viewpoint to make emotional or narrative intent legible.Changing one component commits the other two; for example, a looming viewpoint depends on an upright figure.
- Research gap: Existing tools and generative methods typically model pose, camera, or image relighting separately rather than producing a joint editable 3D staging from emotional text.Pose and motion methods output bodies, camera methods condition on motion without light, and relighting methods produce images rather than 3D scenes.
- Contributions: The corpus contains 11,911 emotional text–3D staging pairs reconstructed from paintings and paired with ArtEmis annotator responses.The staging includes jointly recovered pose, lighting, and camera references.
- Contributions: A flow-matching transformer generates pose, lighting, and camera jointly from one sentence while producing several distinct alternatives per query.The evaluation also examines the trade-off between text alignment and variation.
2 Dataset
The dataset is built by filtering figurative paintings, reconstructing SMPL bodies, estimating low-frequency lighting and camera parameters, and pairing retained scenes with ArtEmis descriptions. This process yields 2,328 paintings and 11,911 description–staging pairs.
- Selection: 6,826 paintings remain after initial filtering for annotator-agreed emotion, reconstruction suitability, people, figure size, and no more than four figures.The initial pool comes from the intersection of ArtEmis and WikiArt, with Mask R-CNN removing unsuitable paintings.
- Reconstruction: SMPL bodies are fitted with HMR2, using projected silhouette agreement with detector masks to score each reconstruction.Each detected person box is processed separately.
- Reconstruction: Low-frequency illumination is estimated by fitting first-order spherical harmonics to luminance samples near each head.The fitted coefficients provide dominant light direction, magnitude, directional variation, chromaticity, and correlated colour temperature.
- Reconstruction: Camera reconstruction places the group’s lowest vertex on the ground, fixes field of view at 45°, and maps multiple figures into shared full-image coordinates.The conversion uses 4DHumans crop-to-full coordinates.
- Dataset scale: 2,328 paintings survive reconstruction filtering, producing 11,911 description–staging pairs from up to six ArtEmis sentences per work.Silhouette IoU and lighting fit account for almost all rejected candidates.
3 Method
The method represents variable-sized scenes as typed pose, light, and camera tokens and uses a text-conditioned flow-matching transformer to generate their joint staging. Guidance at sampling time controls the alignment–variation trade-off.
- Representation: A staging contains up to four typed figure, light, and camera vectors, allowing scenes with different figure counts without padding.Typed tokens also let figures attend to one another, while wrapped angular quantities use suitable periodic representations.
- Model: The transformer receives the CLIP-encoded description as a token sequence through cross-attention, allowing camera and light tokens to attend to relevant words.The architecture uses four pre-norm transformer layers at width 256.
- Objective: Flow matching trains the network to regress the endpoint displacement along interpolated noise-to-scene trajectories and samples outputs in 30 Euler steps.Dropping descriptions during training enables classifier-free guidance at sampling time.
- Guidance: Increasing guidance weight improves text alignment while reducing the number of alternatives per description.The model surpasses CLIP retrieval near w≈2.5 and approximately matches corpus-level diversity at w=6.
4 Evaluation
The evaluation measures retrieval alignment, corpus-level diversity, and multimodality against CLIP-space retrieval. The model improves retrieval while retaining repeated-sample variation and approximately preserving corpus-level diversity.
- Metrics: Evaluation measures R@1 alignment on 32 held-out descriptions, diversity across descriptions, and multimodality across repeated samples.The retrieval baseline selects the nearest training caption in frozen CLIP space and has zero multimodality by construction.
- Results: 32.2% R@1 versus 16.6% for retrieval, with 0.589 multimodality and approximately matched corpus-level diversity at guidance weight w=6.The result comes from the classifier-free guidance ablation.
5 Discussion and Future Work
The corpus and staging model have important scope limitations: affective descriptions rarely supervise individual staging components, and paintings overrepresent portraits. Future work targets broader distributions and confidence-aware training.
- Affective descriptions usually omit pose, lighting, and framing, limiting direct supervision for each staging component.
- The corpus is dominated by portraits, covering a narrow range of poses.
- Future work could weight training by lighting confidence and use film data to broaden pose and shot distributions.