SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction

1 Introduction

• Single-image clothed-human reconstruction must capture body shape and garments’ diverse topology, geometry, and dynamics.The task has applications in AR/VR, virtual try-on, gaming, and film production [33].Source: §1 · p. 2
• SMPL-conditioned multiview methods can produce incorrect topologies and mismatched details when pose estimates are inaccurate or loose clothing is poorly modeled.Pose estimation is especially inaccurate under occlusion or challenging poses.Source: §1 · p. 2
• Native 3D generators handle challenging poses but typically produce coarse, low-fidelity shapes that poorly match the input image.Improving their geometric detail and input consistency remains an open challenge.Source: §1 · p. 2
• SyncHuman combines 2D multiview and native 3D generation, using generated 3D shapes to guide multiview images and detailed images to improve 3D shape fidelity.The framework uses 2D-3D synchronization attention to align the branches and a multiview guided decoder to inject pixel-aligned image information, improving geometric detail and texture fidelity.Source: §1 · pp. 2–3
• SyncHuman outperforms prior single-view reconstruction methods [62] [13] [25] and achieves higher fidelity and texture quality than large-scale 3D generative models [61].The compared 3D models were trained with datasets hundreds of times larger than SyncHuman’s.Source: §1 · p. 3

2 Related works

• Earlier single-image reconstruction used explicit methods that often sacrifice local detail and implicit methods that improve continuous geometry recovery but remain limited in photorealistic appearance and detail.PIFu [45] [6] established pixel-aligned implicit functions; later work added body models, normal or depth supervision, transformers, and larger datasets.Source: §2 · p. 3
• Multiview generation methods synthesize consistent images before optimization-based or feed-forward reconstruction, whereas native 3D models generate directly in 3D representation spaces.Native 3D models are commonly trained on extensive 3D datasets.Source: §2 · p. 3
• Generative human reconstruction includes SDS-based per-case mesh optimization and pose-guided back-view prediction, while sparse multiview methods still struggle with cross-view consistency.Human3Diff adds 3D representations as intermediate constraints, but 2D denoising methods can produce anatomically implausible structures without a body prior.Source: §2 · pp. 3–4
• SyncHuman aligns pretrained 2D multiview and native 3D generators to produce geometrically consistent human models without relying on a human prior.Its overview generates multiview color and normal maps alongside an aligned sparse voxel grid, then uses generated images to guide decoding into a textured mesh.Source: §2 · p. 4, Figure 3 · p. 4

3 Method

• Given one color image, SyncHuman generates multiview color and normal maps with an aligned sparse voxel grid, then decodes these into a detailed textured mesh.The structured latents are produced from the aligned grid using a pretrained flow transformer.Source: §3 · p. 4

3.1 2D-3D Cross-Space Generative Model

• The 2D branch generates four orthogonal views, while a native 3D branch supplies structural guidance without relying on estimated SMPL meshes.The 3D model produces sparse voxel structure; the 2D model generates color and normal maps, using the input as the front view.Source: §3.1 · pp. 4–5
• Each voxel feature attends to corresponding four-view features, while each 2D feature attends to a voxel column, mutually refining the branches.This synchronization aligns the models in shared 3D space and supports multiview consistency and shape fidelity.Source: Figure 4 · p. 5, §3.1 · p. 5
• The cross-space model is jointly trained with flow matching, and the multiview model is retargeted from Stable Diffusion 2.1 to Trellis’s flow-matching setup.This allows the two models to be trained jointly.Source: §3.1 · pp. 5–6

3.2 Multiview Guided Decoder (MVGD)

• MVGD concatenates each voxel’s structured latent with corresponding four-view DINOv2 features before decoding meshes or 3D Gaussian splats.The feature injection is intended to recover detail such as faces and clothing wrinkles; rendered 3D Gaussian splats are baked onto the mesh for the final textured output.Source: §3.2 · p. 6

4.1 Experiment Setup

• The evaluation uses 100 X-Humans scans and 150 CAPE scans, with CAPE divided into 50 CAPE-FP and 100 CAPE-NFP samples.Geometry is assessed with P2S, Chamfer Distance, and Normal Consistency; appearance uses PSNR, SSIM, and LPIPS across four rendered views.Source: §4.1 · p. 6

4.2 Comparison with baseline methods

• Across CAPE-NFP, CAPE-FP, and X-Humans, SyncHuman achieves superior performance on all reported geometry and appearance metrics versus baseline methods.Qualitative comparisons also report improved structural integrity, detail, and natural appearance, including for complex poses and novel viewpoints.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Ablation Study

• On the X-Humans subset, 2D–3D synchronization attention improves geometric accuracy and texture fidelity over PSHuman and Trellis.The comparison removes synchronization attention and fine-tunes models on the same dataset.Source: Figure 7 · p. 8, §4.3 · p. 9, Table 2 · p. 9
• Multiview-guided decoding significantly enhances geometric accuracy and texture quality over the original and human-scan-tuned Trellis decoders.The decoder comparison uses the same structured latents and evaluates geometry and appearance on the X-Humans subset.Source: Figure 8 · p. 8, §4.3 · p. 9, Table 3 · p. 9
• Against 4D-Humans SMPL estimates, SyncHuman’s generated structure aligns better with the input in some complex poses and avoids self-intersection artifacts.Figure 9 compares both structures given the same input image.Source: §4.3 · p. 9, Figure 9 · p. 9

5 Limitation and Conclusion

• Training images rendered with uniform lighting can lead to texture artifacts under extreme lighting, including localized overexposure or shadows.Figure 10 illustrates unnatural textures under non-uniform lighting.Source: §5 · p. 10, Figure 10 · p. 10
• The multiview model’s generation quality remains constrained after fine-tuning SD 2.1 on approximately 5,000 human scans.Source: §5 · p. 10
• The authors identify video generative models and large-scale multiview human datasets as promising directions for scaling the model.Source: §5 · p. 10

A.1 Training Details

• The cross-space model is trained for 50,000 iterations with batch size 32, initialized from the fine-tuned multiview branch and Trellis, with zero-initialized synchronization-attention outputs.The multiview branch is trained separately for 30,000 iterations with batch size 32; training randomly omits image conditioning at rate 0.05 for classifier-free guidance.Source: §A.1 · p. 16
• The multiview-guided decoder is trained for 14,000 iterations on one H800 GPU, using batch size 4 and learning rate 1e-4.Source: §A.1 · p. 16
• The Gaussian-splat decoder objective combines reconstruction, volume, and opacity losses; reconstruction uses L1, SSIM, and LPIPS, while regularization discourages oversized or transparent Gaussians.For each active voxel, K Gaussians are predicted, with scale and opacity denoted by s and α.Source: §A.1 · p. 16
• The mesh objective includes geometry, color, and regularization losses, with regularization terms for attribute consistency, deviation, and TSDF.The consistency term penalizes variance among attributes associated with the same voxel vertex.Source: §A.1 · p. 16

A.2 Detailed Network Structure

• The 3D branch receives two 2D-to-3D attention blocks after transformer blocks 8 and 16, while the 2D branch receives two 3D-to-2D blocks following the 3rd CrossAttnDownBlockMV2D and UpBlock2D modules.Source: §A.2 · p. 16
• Each 3D voxel feature is orthographically projected onto front, back, left, and right view-normal map planes to obtain 2D pixel features; voxel and pixel features then undergo MLP transformations.The projected pixel features are transformed into keys and values by separate MLPs.Source: §A.2 · pp. 16–17
• For 3D-to-2D attention, each pixel queries voxel features sampled along its 3D ray, and the attention output updates the pixel feature.The sampled sequence contains H voxel features; the pixel feature supplies the query, while voxel features supply keys and values.Source: §A.2 · p. 17
• For upsampled multiview color and normal images, DINOv2 features from layers 4, 11, 17, and 23 are concatenated and passed through an MLP; voxel positions project to view pixels for direct feature lookup.Source: §A.2 · p. 17

B.1 More Results

• LHM and IDOL rely on SMPL, so inaccurate or failed SMPL estimation yields erroneous structure; both also show limited fidelity.Hunyuan3D 2.5 produces reasonable structures with details but its meshes have less fidelity; SyncHuman is described as visually high-fidelity.Source: §B.1 · p. 18
• After 2D-3D attention alignment, projections from the two branches can almost completely overlap.Figure 12 illustrates the aligned multiview projections.Source: Figure 12 · p. 19
• Figure 13 compares intermediate multiview generation quality with and without 2D-3D attention.The caption identifies the comparison but does not state its outcome.Source: Figure 13 · p. 19

B.2 Ablation of the quality of intermediate multi-view generation and 3D structure generation

• 0.5907 vs. 0.4813 3D-structure IoU with vs. without 2D-3D synchronization attention on a small human-scan subset.The authors also report that attention makes multiview images more structurally reasonable and the two branches' projections almost completely overlap.Source: §B.2 · p. 19
• The decoded mesh may contain surface holes in some cases.Figure 15 illustrates this possibility.Source: Figure 15 · p. 20

B.3 Inference time.

• 38.57s vs. 15.68s vs. 52.98s inference time for SyncHuman, Trellis, and PSHuman, respectively, on a single H800.SyncHuman is faster than PSHuman because it directly decodes a 3D shape without differentiable rendering optimization, and slower than Trellis due to multiview generation.Source: §B.3 · p. 20

B.4 Unconditional Generation.

• Unconditional generation quality is worse than generation conditioned on a single-view image.Figure 14 shows the unconditional-generation task.Source: §B.4 · p. 20, Figure 14 · p. 20

C.1 Limitations about containing holes in the generation

• FlexiCube in the Trellis mesh decoder imposes no watertight-surface constraint, so holes may appear in some generated meshes.The authors suggest SDF fitting or using an SDF-target native 3D model as possible future approaches.Source: §C.1 · p. 20

C.2 Differences from SyncDreamer.

• Unlike SyncDreamer, SyncHuman synchronizes a 2D generative model with a native 3D model, and its volume is a 3D representation generated from noise.SyncHuman injects information in both directions, 2D-to-3D and 3D-to-2D.Source: §C.2 · p. 21

C.3 Ethics Statement

• The authors identify potential misuse of generated models to deceive viewers, including adult content, political manipulation, and exploitation through digital replicas.The model was trained on THuman2.1, CustomHumans, THuman3.0, and 2K2K, and tested on X-Humans and CAPE.Source: §C.3 · p. 21

• The authors state that the introduction and abstract accurately reflect the paper’s contributions and scope, supported by experimental results.Source: NeurIPS Paper Checklist · p. 22
• The authors identify training-data constraints: uniform-light renders may cause texture artifacts under extreme lighting, and generation quality remains constrained by fine-tuning on about 5,000 human scans.Source: NeurIPS Paper Checklist · p. 22
• The evaluation uses public datasets; code is absent from the supplement but is planned for public release after preparation.Source: NeurIPS Paper Checklist · p. 23
• Quantitative evaluations are reported in Tables 1–3, and training and test details are discussed in the experiments section and supplement.Source: NeurIPS Paper Checklist · p. 24
• The paper reports ethics-code conformity, a broader-impacts statement, and planned safeguards for model access and sensitive content.Source: NeurIPS Paper Checklist · p. 25
• The authors report no crowdsourcing or human-subject research, no new assets, and no important, original, or non-standard LLM components.Source: NeurIPS Paper Checklist · pp. 26–27