Source-linked AI summary
PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction
Xiang Zhang, Sohyun Yoo, Hongrui Wu, Chuan Li, Jianwen Xie, Zhuowen Tu
TL;DR
Single-view indoor scene reconstruction is ill-posed because images provide partial, depth-ambiguous observations with occluded regions. PixARMesh combines pixel-aligned image features and global scene context with autoregressive mesh generation to jointly predict object poses and meshes. It achieves competitive object-level accuracy and state-of-the-art scene-level performance while producing compact, artist-ready meshes.
Problem
Single-view reconstruction must recover complete scene geometry and layout from partial, depth-ambiguous RGB observations, while existing pipelines rely on limited SDF representations or post-hoc layout processing.
Method
PixARMesh augments object-level autoregressive mesh generators with pixel-aligned image features and global scene context, jointly decoding each object’s pose and mesh in one sequence.
Results
PixARMesh achieves competitive object-level accuracy and state-of-the-art scene-level performance while generating compact, artist-ready meshes.
Takeaways & Limitations
Autoregressive mesh generation is presented as a viable alternative to conventional SDF-based scene-reconstruction pipelines for real-world inputs.
Takeaways & Limitations
SDF-based prior pipelines can produce overly smooth, high-face-count meshes, while complex real-world scene reconstruction remains challenging for earlier methods.
Abstract
from arXiv · showhide
We introduce PixARMesh, a method to autoregressively reconstruct complete 3D indoor scene meshes directly from a single RGB image. Unlike prior methods that rely on implicit signed distance fields and post-hoc layout optimization, PixARMesh jointly predicts object layout and geometry within a unified model, producing coherent and artist-ready meshes in a single forward pass. Building on recent advances in mesh generative models, we augment a point-cloud encoder with pixel-aligned image features and global scene context via cross-attention, enabling accurate spatial reasoning from a single image. Scenes are generated autoregressively from a unified token stream containing context, pose, and mesh, yielding compact meshes with high-fidelity geometry. Experiments on synthetic and real-world datasets show that PixARMesh achieves state-of-the-art reconstruction quality while producing lightweight, high-quality meshes ready for downstream applications.
1. Introduction
Single-view indoor scene reconstruction is ill-posed because RGB images provide partial, depth-ambiguous observations and leave much of the scene occluded. PixARMesh addresses limitations of SDF- and optimization-based pipelines by jointly predicting poses and native meshes autoregressively.
- Motivation: Single-view reconstruction must infer complete object geometry and spatial layout from partial, depth-ambiguous observations with substantial occlusion.Accurate recovery therefore depends on priors about indoor scenes and plausible object structures.
- Prior limitations: SDF-based holistic methods are constrained by volumetric resolution and decoder expressiveness, producing lower-quality geometry and limiting generalization to complex real-world scenes.SDF representations also add surface-extraction complexity and can yield overly smooth, high-face-count meshes.
- Prior limitations: Compositional pipelines commonly require amodal completion for occluded objects and optimization-based layout estimation to assemble reconstructed instances.Their layout estimation is often formulated as point-cloud matching.
- Motivation: Object-level autoregressive mesh generators directly predict artist-like mesh sequences, but prior scene reconstruction pipelines did not exploit these native representations.This gap motivates combining partial observations with mesh-level generative priors for scene reconstruction.
- PixARMesh: PixARMesh fuses pixel-aligned image features and global scene context with point-cloud features, then jointly predicts object poses and meshes in one autoregressive sequence.The framework is built on pre-trained object-level mesh generators and replaces complex optimization loops with feed-forward decoding.
2. Related Work
Prior scene reconstruction methods range from holistic volumetric prediction to compositional instance reconstruction, while native mesh generation has advanced through autoregressive and compressed tokenizations. PixARMesh combines these directions in a scene-level autoregressive framework.
- Single-view scene reconstruction: Holistic single-view methods reconstruct scenes from back-projected feature volumes but face spatial-resolution limits and poor generalization.The task remains ill-posed because of scale ambiguity, occlusion, and incomplete geometric cues.
- Compositional generation: Compositional methods reconstruct individual instances using retrieval, procedural assembly, inpainting, or object-level generative models before composing the scene.Some approaches complete occluded regions through image inpainting before reconstruction.
- Native mesh generation: Native mesh generation targets compact, editable, downstream-compatible geometry, evolving from structured primitives toward discrete diffusion and autoregressive models.Earlier primitive-based methods impose strong geometric priors and limited topological flexibility.
- Native mesh generation: MeshGPT and related models generate tokenized meshes autoregressively, while MeshXL and later methods improve coordinate-space operation, compression, and scalability.Compression strategies exploit face adjacency, and Meshtron internally compresses long sequences with an Hourglass Transformer.
- PixARMesh: PixARMesh’s pipeline extracts depth point clouds and image features, aggregates local and global cues, and autoregressively predicts pose followed by mesh tokens.This connects object-level mesh generation to scene-level reconstruction.
- Native mesh generation: Complementary work improves mesh quality or controllability through reinforcement learning, coarse structural guidance, and coarse-to-fine detail generation.PixARMesh builds on compressed and scalable object-level models such as EdgeRunner and BPT.
3. Method
PixARMesh adapts object-level autoregressive mesh generators for single-view scene reconstruction by combining partial geometry, pixel-aligned appearance, and global scene context. A unified token sequence predicts each object’s pose and canonical mesh, which are transformed into the scene frame.
- Problem setup: The method reconstructs foreground object instances from a single RGB image while disregarding large planar background structures such as walls and floors.This follows a compositional scene-reconstruction setup focused on indoor foreground objects.
- Scene-level formulation: PixARMesh jointly predicts each object’s scene-level pose and canonical mesh instead of reconstructing shapes first and resolving layout afterward.The final scene transforms each canonical mesh into the global coordinate frame.
- Problem setup: Off-the-shelf perception models provide depth, instance masks, and image features; depth is back-projected into global camera-coordinate point clouds and masked per object.Each object point cloud contains only its visible portion.
- Pixel-aligned encoding: The point-cloud encoder fuses geometric features with image features sampled at projected pixel locations through a Transformer-based fusion block.Learnable queries aggregate the fused per-point features into a compact latent code.
- Scene context: Global scene context is preserved by normalizing instance and scene point clouds in one coordinate frame, then applying cross-attention from each object latent to the scene latent.This supplies cues from nearby objects and supports context-aware reconstruction under missing geometry.
- Tokenization: Pose and mesh outputs are represented as discrete tokens, with pose decoding producing eight normalized scene-space bounding-box corners.The mesh sequence is decoded in canonical unit-cube space.
- Tokenization: A best-fit affine transform maps canonical box corners to decoded global corners and is applied to every canonical-mesh vertex.The transform is estimated by linear least squares and interpreted as gravity-aligned.
- Mesh tokenization: BPT and EdgeRunner retain their native mesh tokenizations, respectively emphasizing aggressive compression or geometric detail with different vocabularies and compression ratios.The framework supports both tokenization paradigms without modification.
4. Experiments
PixARMesh is evaluated on synthetic and real-world indoor scenes using geometric metrics, qualitative comparisons, and targeted ablations. It achieves strong object- and scene-level reconstruction while producing compact meshes, and the ablations identify the roles of joint modeling, image features, context, and upstream perception.
- Settings: Experiments use 3D-FRONT for training and synthetic evaluation, plus Pix3D, Matterport3D, and ScanNet to assess real-world generalization.The synthetic test subset contains 100 object-level and 156 scene-level evaluation scenes.
- Settings: Evaluation measures object-level geometric fidelity and scene-level reconstruction using Chamfer Distance, F-Score, and single-direction Chamfer Distance.Chamfer Distance is lower-is-better, F-Score is higher-is-better, and CD-S accounts for missing instances.
- Main Results: At the object level, PixARMesh achieves second-best performance with F-Score comparable to diffusion-based SDF models while producing compact meshes with only a few thousand faces per instance.Unlike SDF-based pipelines, it directly generates meshes without Marching Cubes surface extraction.
- Main Results: At the scene level, PixARMesh achieves state-of-the-art performance across all reported metrics by jointly predicting object geometry and pose with image and scene context.Qualitative results show coherent shapes and spatial arrangements, with clear edges, structural boundaries, and smooth surface continuity.
- Main Results: On real-world images, PixARMesh shows reasonable generalization and reconstructs indoor environments with practical, interpretable geometry despite primarily synthetic training.Additional qualitative results report stronger layout alignment and coherent reconstructions across diverse environments and lighting conditions.
- Ablation Studies: Joint pose-mesh modeling outperforms post-hoc optimization and two-stage prediction, while pixel-aligned image features and scene context improve complementary aspects of reconstruction.Joint modeling improves object CD from 4.75 to 4.04 and F-Score from 80.85 to 82.27; removing image features causes the largest ablation drop.
5. Conclusion
PixARMesh reconstructs indoor scenes autoregressively in mesh space, jointly predicting object pose and geometry without SDFs or post-hoc layout optimization. It achieves competitive object-level accuracy, state-of-the-art scene-level performance, and compact artist-ready meshes.
- PixARMesh jointly predicts object pose and geometry for coherent full-scene reconstruction without SDFs or post-hoc layout optimization.
- PixARMesh achieves competitive object-level accuracy and state-of-the-art scene-level performance.
- The method generates compact, artist-ready meshes and demonstrates applicability to real-world inputs.
A.1.1. Data Pre-processing
The data-preprocessing pipeline adapts 3D-FRONT meshes for autoregressive tokenization through vertex merging, mesh simplification, and quality selection. The processed dataset averages 1,809 faces per shape.
- 3D-FRONT meshes are pre-processed to make them suitable for autoregressive tokenization.
- Nearby vertices are merged using minimum spatial resolutions determined by quantization levels q ∈128, 256, 512, 1024.
- Mesh simplification combines planar decimation with quadric-error-based edge-collapse decimation targeting 800, 2,000, or 4,000 faces.
- Quality selection uses Hausdorff distance to compare simplified meshes with originals under an empirical threshold τ = 0.01.
- 1,809 faces per shape is the average size of the processed dataset meshes.
A.1.2. Training Details
PixARMesh training first bootstraps layout prediction and then jointly trains pose and mesh generation. Both stages use AdamW with scheduled learning-rate decay, batch size 64, and gradient clipping.
- Training uses layout bootstrapping followed by joint pose-mesh training.
- The EdgeRunner variant uses 100k layout iterations and 30k joint iterations, while BPT uses 30k and 25k iterations, respectively.
- Both stages use AdamW with an initial learning rate of 1 × 10−4, 500-step warmup, cosine decay to 1 × 10−5, and effective batch size 64.
- Optimization uses β1 = 0.9, β2 = 0.95, and gradient clipping at 1.0.
A.2.1. More Qualitative Results on Real Images
Additional evaluations show that PixARMesh produces coherent, compact reconstructions on real indoor images, with strong layout accuracy and substantially fewer faces than prior methods. Ablations indicate that image features and scene context improve scene-level alignment across mesh backbones.
- More Qualitative Results on Real Images: PixARMesh generally achieves stronger layout alignment and coherent reconstructions across real indoor scenes and diverse lighting conditions.
- Mesh Compactness: PixARMesh produces around 7–8k faces per scene, compared with roughly 2M for InstPIFu and about 0.3M for BUOL, Gen3DSR, and DeepPriorAssembly.
- Layout Accuracy: PixARMesh achieves higher 3D bounding-box IoU layout accuracy than prior methods relying on predicted depth.
- Layout Accuracy: The EdgeRunner-based variant consistently produces more accurate layouts than the BPT-based model.
- Inference Runtime: PixARMesh is slower than feed-forward and latent-diffusion approaches but generally faster than other compositional pipelines while producing native artist-ready meshes.
- Ablation Studies: Adding pixel-aligned image features substantially improves scene-level accuracy, while scene-context aggregation provides further gains across mesh tokenizations and backbones.