Source-linked AI summary
Pixal3D: Pixel-Aligned 3D Generation from Images
Dong-Yang Li, Wang Zhao, Yuxin Chen, Wenbo Hu, Meng-Hao Guo, Fang-Lue Zhang, Ying Shan, Shi-Min Hu
TL;DR
Image-to-3D generation still struggles to preserve pixel-level fidelity because 2D–3D correspondence remains ambiguous. Pixal3D addresses this with pixel-aligned generation and back-projected image features, substantially improving fidelity while extending to multi-view and scene generation.
Problem
Image-to-3D methods often produce misaligned shapes and lose fine details, limiting faithful visible-surface reconstruction and plausible completion of unseen regions.
Method
Pixal3D generates objects in the input camera frame and uses back-projection to establish explicit pixel-to-3D correspondence through aligned feature volumes.
Results
Pixal3D substantially improves 3D fidelity while supporting scalable single-view, multi-view, and object-separated scene generation.
Takeaways & Limitations
Pixal3D combines image-constrained visible surfaces with learned completion of invisible regions for faithful 3D objects and scenes.
Takeaways & Limitations
The framework is sensitive to pixel-level noise, assumes reasonably accurate camera poses for multi-view inputs, and may err in complex occlusions.
Abstract
from arXiv · showhide
Recent advances in 3D generative models have rapidly improved image-to-3D synthesis quality, enabling higher-resolution geometry and more realistic appearance. Yet fidelity, which measures pixel-level faithfulness of the generated 3D asset to the input image, still remains a central bottleneck. We argue this stems from an implicit 2D-3D correspondence issue: most 3D-native generators synthesize shape in canonical space and inject image cues via attention, leaving pixel-to-3D associations ambiguous. To tackle this issue, we draw inspiration from 3D reconstruction and propose Pixal3D, a pixel-aligned 3D generation paradigm for high-fidelity 3D asset creation from images. Instead of generating in a canonical pose, Pixal3D directly generates 3D in a pixel-aligned way, consistent with the input view. To enable this, we introduce a pixel back-projection conditioning scheme that explicitly lifts multi-scale image features into a 3D feature volume, establishing direct pixel-to-3D correspondence without ambiguity. We show that Pixal3D is not only scalable and capable of producing high-quality 3D assets, but also substantially improves fidelity, approaching the fidelity level of reconstruction. Furthermore, Pixal3D naturally extends to multi-view generation by aggregating back-projected feature volumes across views. Finally, we show pixel-aligned generation benefits scene synthesis, and present a modular pipeline that produces high-fidelity, object-separated 3D scenes from images. Pixal3D for the first time demonstrates 3D-native pixel-aligned generation at scale, and provides a new inspiring way towards high-fidelity 3D generation of object or scene from single or multi-view images. Project page: https://ldyang694.github.io/projects/pixal3d/
1 Introduction
Pixal3D addresses the fidelity bottleneck in image-to-3D generation by replacing ambiguous canonical-space conditioning with explicit pixel-to-3D correspondence. It generates pixel-aligned 3D assets and extends the paradigm to multi-view generation and object-separated scene synthesis.
- Problem: Current image-to-3D methods remain limited by fidelity, often producing approximately similar shapes with misalignment and lost fine details.Fidelity measures how faithfully a generated 3D asset matches its input image.
- Motivation: Reconstruction achieves stronger fidelity because it explicitly establishes 2D-3D correspondence, unlike generation methods that leave this association implicit.The introduction attributes reconstruction’s advantage to correspondence established through pixel correspondences and triangulation.
- Pixal3D: Pixal3D directly generates 3D in a pixel-aligned pose consistent with the input image, replacing cross-attention with back-projected image features in a 3D volume.The back-projection scheme explicitly injects pixel information into 3D and lifts image features along rays.
- Extensions: Pixal3D extends to multi-view generation by averaging back-projected feature volumes from individual views.This provides a shared formulation for single-view and multi-view settings.
- Scene generation: A modular Pixal3D pipeline composes object-level generations into high-fidelity, object-separated 3D scenes.The paper also presents pixel-aligned generation as a paradigm combining image-constrained visible surfaces with generatively completed invisible regions.
2 Related Works
3D generation has progressed from 2D-prior distillation to scalable 3D-native pipelines across diverse representations, but image-to-3D fidelity remains limited by implicit 2D–3D correspondence. Reconstruction offers explicit correspondence but incomplete assets, motivating Pixal3D’s pixel-aligned generation paradigm and the broader combination of reconstruction with generative modeling.
- 3D Generation: 3D generation spans 2D diffusion-prior distillation, 3D-native dataset learning, and representations including points, voxels, meshes, Gaussians, and triplanes.These representations aim to balance fidelity, efficiency, and scalability.
- 3D Generation: Current image-to-3D methods often generate canonical-pose shapes and inject image information through cross-attention, leaving 2D–3D correspondence implicit and reducing pixel fidelity.Pixal3D instead directly generates pixel-aligned 3D objects.
- 3D Reconstruction: Reconstruction methods establish explicit correspondences through SfM, MVS, triangulation, optimization, plane-sweeping, and pixel-aligned prediction of 3D structures.Recent approaches predict depths, normals, point maps, or 3D Gaussians in a pixel-aligned manner.
- 3D Reconstruction: Reconstruction achieves high fidelity but produces incomplete outputs, while Pixal3D applies its explicit correspondence principle to generate complete assets with reconstruction-level fidelity.The contrast motivates pixel-aligned modeling for 3D generation.
- 3D Generative Reconstruction: The complementarity of reconstruction and generation has led to 3D generative reconstruction, combining input consistency with complete and plausible outputs beyond the observed views.Early methods use image generative models to complete insufficient 2D views and enhance reconstruction.
3 Method
Pixal3D performs 3D generation in the input camera’s coordinate frame and establishes explicit pixel-to-3D correspondence through back-projected image features. Its framework extends this conditioning to multi-view aggregation and pixel-aligned object composition for scene synthesis.
- Pixel-Aligned Generation: Pixal3D defines objects in the input camera’s coordinate frame, aligning the generated 3D volume with image pixels and camera rays.This replaces canonical, view-independent object representations whose 2D-3D correspondence is underconstrained for image-conditioned generation.
- Back-Projection Conditioning: The model encodes pixel-aligned objects into 3D latents and learns view-dependent generation, using back-projection rather than cross-attention to inject image information into 3D.Image features are projected along camera rays into a 3D feature volume, which is added directly to the diffusion noise volume; a DINOv2 global token supplies semantic guidance.
- Multi-Scale Features: Multi-scale image features preserve low- and high-level information by upsampling DINOv2 patch tokens, projecting each voxel to the image plane, and averaging sampled features.The resulting high-resolution features improve recovery and consistency of fine details.
- Multi-View Generation: With known camera parameters, Pixal3D extends to multi-view generation by back-projecting features from each view into 3D space and aggregating them.The extension follows directly from the single-view model’s explicit projection geometry.
- Scene Synthesis: For scene synthesis, Pixal3D generates individual objects from a scene image and composes them through image-space alignment into a full 3D scene.Compared with SAM3D, the pipeline avoids estimating a 7-DoF object pose and uses pixel-aligned generation with global depth estimation to improve multi-object alignment accuracy and stability.
4 Experiments
Pixal3D is evaluated against representative single-view, multi-view, and scene-generation baselines using quantitative, qualitative, and ablation studies. Across these evaluations, it improves fidelity and alignment, while remaining subject to noise, camera-pose, and occlusion-completion limitations.
- Single-view evaluation: Pixal3D is quantitatively evaluated against TRELLIS, TripoSG, Hunyuan3D-2.1, and Direct3D-S2 on Toys4K using rendered-normal fidelity comparisons.The evaluation uses ground-truth camera poses for baselines, while Pixal3D leverages its inference-time projection.
- Single-view evaluation: The evaluation measures normal-map overlap, pixel discrepancy, angular errors, boundary errors, and thresholded angular accuracy on overlapping regions.The metrics are IoU, PSNR, mean and median angular error, Mean_B, and accuracy under different angular thresholds.
- Single-view evaluation: Pixal3D achieves substantial improvements across all reported fidelity metrics and more faithfully recovers fine-grained visual details than competing methods.Qualitative comparisons highlight keyboard layouts, facial details, and flower-petal number and arrangement; an additional 150-image test set uses ULIP2, Uni3D, and a 30-participant user study.
- Multiview evaluation: Across 2, 4, and 6 input views, Pixal3D significantly outperforms VGGT and multiview TRELLIS on Chamfer Distance, Earth Mover’s Distance, and F-Score.Qualitatively, Pixal3D avoids the floaters, outliers, cross-view inconsistencies, and hallucinations observed in the baselines.
- Scene generation and ablations: Pixel-aligned scene generation produces more consistent inter-object relations than SAM3D, while feature upsampling and back-projection conditioning are necessary for stable, high-fidelity generation.Removing feature upsampling limits detail representation, whereas replacing back-projection with conventional cross-attention slows and destabilizes convergence and lowers fidelity.
- Limitations: Pixal3D remains sensitive to pixel-level noise, assumes known and reasonably accurate multiview camera poses, and relies on 2D inpainting for occluded scene regions.Back-projected noise can create geometric artifacts, while inpainting may occasionally introduce errors in completed regions.
5 Conclusion
Pixal3D presents pixel-aligned 3D generation for high-fidelity asset creation from images, directly aligning generated 3D models with input images through explicit geometric 2D-3D correspondence.
- Pixal3D introduces a pixel-aligned 3D generation paradigm for high-fidelity 3D asset creation from images.
- Unlike canonical-space 3D-native methods, Pixal3D directly creates 3D models aligned with images.
- Its back-projection image conditioning replaces ambiguous cross-attention with explicit geometric 2D-3D correspondence for high-precision, pixel-aligned synthesis.