Source-linked AI summary
An Evaluation Framework for Generating Multi-View Images of a Person in a Scene
Mahir Majid, Young Kyung Kim, Guillermo Sapiro
TL;DR
DiTs lack specialized paired multi-view human-scene data and therefore struggle with spatially consistent camera changes. The paper evaluates synthetic viewpoint editing, introduces HSRD to separate camera motion from head movement, and demonstrates its use for filtering synthetic data. HSRD supports curating geometrically cleaner multi-view datasets, although camera adapters and HSRD still have documented failure cases.
Problem
DiTs struggle with spatially consistent camera changes because paired multi-view datasets of humans in natural scenes are absent and difficult to capture.
Method
The paper evaluates synthetic human-scene viewpoint edits and introduces HSRD, which decouples camera orbit movement from localized head pose manipulation.
Results
HSRD is demonstrated as a quality filter for synthetic multi-view data, while the camera LoRA alters backgrounds more than baseline configurations.
Takeaways & Limitations
HSRD provides a pipeline for filtering synthetic images to curate geometrically cleaner multi-view datasets of people in scenes.
Takeaways & Limitations
The camera LoRA remains prone to geometric hallucinations, and HSRD can produce a false zero when head and scene turns are both zero or the requested direction is reversed.
Abstract
from arXiv · showhide
Recent generative image-editing Diffusion Transformers (DiTs) demonstrate impressive semantic editing capabilities but still struggle with spatially consistent camera angle changes. A primary bottleneck in training foundation models to execute free-form, promptable camera angle changes is the lack of specialized training data. While multi-view datasets exist for generic 3D environments and objects, there remains an absence of paired, multi-view datasets featuring human subjects at fixed locations in natural scenes, including frontal and side-profile views. Capturing such multi-camera data in unconstrained environments is logistically challenging and unscalable. In this paper, we first experiment with multiple state-of-the-art image editing models to create this data synthetically, but find that the outputs are frequently prone to hallucinations involving how much the subject's head turns relative to the background, often producing inconsistent environments. To address this issue, we propose the Head Scene Rotation Difference (HSRD) metric to quantitatively evaluate camera movements around a person. The proposed metric operates by decoupling camera movement from localized head pose manipulation. As demonstrated by the extensive experimentation, HSRD provides the pipeline necessary to evaluate 3D spatial parallax for a person in a scene, paving the way to reliably construct high-quality multi-view synthetic datasets.
1. Introduction
Generative image-editing DiTs support semantic editing but struggle with geometrically consistent camera changes because they lack explicit 3D priors and specialized multi-view data. The paper proposes HSRD to distinguish valid camera orbits from localized head-turn hallucinations and filter synthetic training data.
- DiTs struggle with spatially rigorous novel-view synthesis because their architectures lack explicit 3D geometric priors.
- Large-angle camera prompts often leave the background rigid while unnaturally turning the subject’s head.
- Head-pose analysis alone is insufficient because evaluation must jointly track head orientation and surrounding scene geometry.
- HSRD quantitatively evaluates camera movements by decoupling background scene orbits from localized head movement.
- HSRD is demonstrated as a quality-filtering mechanism for synthetic multi-view datasets.
2. Related Work
Existing datasets provide either scene geometry without dynamic humans or accurate human geometry under controlled capture, while synthetic camera adapters offer a scalable alternative. However, these adapters remain vulnerable to geometric hallucinations, motivating a dedicated filtering pipeline.
- Foundation models need strictly paired multi-view data to learn spatial camera orbits rather than statistical angle approximations.
- Scene-level datasets provide camera matrices and dense views but omit dynamic humans, whereas human-centric datasets use controlled synchronized camera rigs.
- A camera-angle LoRA offers a scalable way to generate missing paired human-scene views when real-world capture is logistically prohibitive.
- At scale, the adapter still often turns only the head while freezing the background or failing to produce a consistent scene change.
- Because raw adapter outputs cannot be blindly trusted, synthetic training pairs require a filtering pipeline that separates global camera orbits from localized head turns.
3. Measuring Viewpoint Shift
The study builds a 40-image indoor human-scene benchmark, evaluates viewpoint edits through background and perceptual changes, and shows that the camera LoRA alters backgrounds more than baseline models. These global metrics reveal attempted shifts but cannot verify geometrically valid 3D rotations.
- Dataset Generation: The benchmark contains 40 frontal reference images spanning eight gender-setting categories across four indoor residential scenes.Five generation seeds are used for each category.
- Viewpoint Shift Generation: The evaluation tests multiple image-editing models with directional prompts, producing 1040 generated image pairs from the 40 starting images.
- Quantitative Evaluation: Background PSNR, MSE, and LPIPS are computed using SAM 3 foreground masks to assess pixel-level and global changes.
- Quantitative Evaluation: A valid camera rotation should lower PSNR while increasing MSE and LPIPS; localized head turns produce the inverse trend.
- Quantitative Results: The camera LoRA produces more background modification than every tested baseline configuration, with average MSE over four times higher than the comparison configurations.
- Quantitative Results: 12.25% of camera LoRA generations remain within one standard deviation of a base-model metric distribution and exhibit background-freezing failures.
- Metric Limitations: Global image metrics cannot distinguish arbitrary pixel distortion from a mathematically valid 3D background rotation.
4. Methodology: HSRD Evaluation Protocol
The HSRD protocol evaluates novel-view synthesis by separating localized head motion from global scene-camera motion. It combines foreground masking, head-pose estimation, and background-based camera-pose estimation to quantify whether an edit represents a valid camera orbit.
- Existing 3D consistency metrics cannot distinguish localized head manipulation from background geometry because they compute one fused score over the unmasked frame.
- HSRD is a model-independent three-stage pipeline comprising absolute head-pose extraction, background-isolated camera-pose extraction, and decoupled rotation-difference calculation.
- Decoupled Rotation Difference: The protocol estimates head yaw change with 6DRepNet and extracts scene yaw from the relative camera rotation, focusing on horizontal azimuth under negligible pitch and roll changes.
- Background-Isolated Camera Pose Estimation: SAM 3 generates binary person masks for both views, and the protocol blacks out foreground human pixels before camera estimation.
- Background-Isolated Camera Pose Estimation: Foreground suppression removes subject features that could bias reconstruction while preserving the image token grid used by VGGT.
- Background-Isolated Camera Pose Estimation: VGGT maps the masked image pair to absolute camera rotations in a shared coordinate system, which are converted into a relative rotation between reference and edited views.
- Decoupled Rotation Difference: HSRD compares head and scene yaw differences to identify whether a generated viewpoint reflects true camera rotation or compensatory head turning.
5. Experimental Results
Experiments validate HSRD by testing foreground masking for camera-pose estimation and applying the metric to filter synthetic human-scene views. Low HSRD corresponds to coherent camera orbits, while high scores expose frozen backgrounds and isolated head rotation.
- 5.1. Validating HSRD: Foreground masking preserves reliable background pose estimation, supporting HSRD’s strategy of excluding dynamic human pixels from scene tracking.VGGT remains reliable under masking, and its masked performance exceeds MASt3R and DUSt3R on the unmasked baseline.
- 5.2. HSRD Filtering for Multi-View Human Scene Data Synthesis: HSRD exhibits high variance across the synthetic dataset, indicating substantial spatial entanglement in generated camera changes.Table 3 reports the overall distribution and directional subsets, including a skew toward higher rotational discrepancy in right-ward prompts.
- 5.2. HSRD Filtering for Multi-View Human Scene Data Synthesis: Low HSRD scores near Q1 align with geometrically logical camera orbits and correctly repositioned scene objects.Qualitative examples show consistent relative placement of beds, pillows, paintings, windows, and counters after requested viewpoint shifts.
- 5.2. HSRD Filtering for Multi-View Human Scene Data Synthesis: HSRD ≤20° is established as a reasonable threshold for isolating valid novel-view synthesis from incomplete or inconsistent camera orbits.Near-median examples can show passable head changes but incomplete spatial rotation, motivating the threshold.
- 5.2. HSRD Filtering for Multi-View Human Scene Data Synthesis: High HSRD scores near Q3 identify background-freeze hallucinations in which the head turns substantially while the scene barely rotates.The score rises because HSRD detects approximately zero background rotation alongside significant head rotation.
- 5.2. HSRD Filtering for Multi-View Human Scene Data Synthesis: 74.25% of generated images achieved at least a 30° head rotation in the requested direction, while HSRD filtering retained 169 valid pairs from 400 generations.Of 176 images below the 20° HSRD threshold, 169 also passed the head-pose criterion; 7 were False Zero outliers.
6. Conclusion
HSRD supports filtering synthetic multi-view images to isolate valid camera orbits from localized head-turn failures. Its current scope is a single primary human subject, with extension to multiple subjects and background people left for future work.
- HSRD can filter synthetic images with high scores to curate geometrically clean multi-view datasets of people in scenes.The proposed pipeline is intended to support automatic curation from large volumes of generated images.
- The methodology is designed for scenes containing a single primary human subject.
- Extending HSRD to multiple primary subjects and people in the background remains future work for real-world cinematic validation.