Source-linked AI summary
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, Kai Zhang
TL;DR
PF-LRM addresses sparse, wide-baseline unposed reconstruction, where conventional SfM and pose-dependent methods become unreliable. It jointly predicts 3D shape and camera poses using cross-modal transformer attention and differentiable PnP, outperforming baselines across unseen datasets.
Problem
Sparse, wide-baseline images undermine SfM and methods requiring accurate camera poses, limiting reliable 3D reconstruction from few unposed views.
Method
PF-LRM uses self-attention between 2D image and 3D NeRF tokens, predicts per-view coarse point clouds, and solves camera poses with differentiable PnP.
Results
PF-LRM outperforms baseline methods by a large margin in pose prediction accuracy and 3D reconstruction quality across unseen evaluation datasets.
Takeaways & Limitations
The model supports robust reconstruction from 2–4 unposed images and downstream text/image-to-3D generation.
Takeaways & Limitations
PF-LRM ignores background cues for pose prediction and cannot model view-dependent appearance effects.
Abstract
from arXiv · showhide
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images even with little visual overlap, while simultaneously estimating the relative camera poses in ~1.3 seconds on a single A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention blocks to exchange information between 3D object tokens and 2D image tokens; we predict a coarse point cloud for each view, and then use a differentiable Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge amount of multi-view posed data of ~1M objects, PF-LRM shows strong cross-dataset generalization ability, and outperforms baseline methods by a large margin in terms of pose prediction accuracy and 3D reconstruction quality on various unseen evaluation datasets. We also demonstrate our model's applicability in downstream text/image-to-3D task with fast feed-forward inference. Our project website is at: https://totoro97.github.io/pf-lrm .
1 INTRODUCTION
PF-LRM addresses sparse, wide-baseline reconstruction by jointly estimating camera poses and 3D shape from 2–4 unposed images. Its transformer exchanges image and NeRF information, predicts coarse surface points for differentiable PnP, and generalizes across datasets.
- Motivation: Sparse, wide-baseline views often make SfM unreliable, preventing pose-dependent neural reconstruction methods from being used reliably.SfM assumes dense viewpoints, sufficient overlap, and matching features.
- Approach: PF-LRM jointly predicts relative camera poses and object shape and appearance from as few as 2–4 unposed images.The object representation is a triplane NeRF.
- Approach: Self-attention over 2D image and 3D triplane tokens enables information exchange for NeRF reconstruction and pose estimation.Final NeRF tokens represent the triplane NeRF, while contextualized image patch tokens predict coarse point clouds.
- Approach: PF-LRM converts pose prediction into per-patch 3D surface-point prediction, then uses a differentiable Perspective-n-Point solver for camera estimation.The predicted points correspond to 2D patch centers and receive online supervision from NeRF geometry.
- Results: ∼590 million parameters and training on ∼1 million objects support variable 2–4-view testing and strong cross-dataset results.The model is trained on posed Objaverse renderings and real MVImgNet captures without direct 3D supervision.
- Results: PF-LRM achieves state-of-the-art pose estimation and novel view synthesis for very sparse inputs, outperforming baselines by a large margin.The paper also demonstrates downstream text/image-to-3D applications.
2 RELATED WORK
Prior sparse-view reconstruction and pose methods generally assume accurate poses, sufficient overlap, or staged refinement. PF-LRM instead jointly reconstructs NeRFs and estimates poses in one scalable pipeline using predicted 3D–2D correspondences.
- Sparse-view NeRF: Sparse-view NeRF methods typically assume precise camera poses for every input image.Earlier NeRF required hundreds of posed images, while later methods use regularization or learned priors.
- Structure from Motion: SfM estimates poses and sparse 3D structure from feature matches, but requires sufficient visual overlap and discriminative features.Extremely sparse views make these conditions difficult to satisfy.
- Neural pose prediction: Direct neural pose regressors omit 3D shape information, whereas PF-LRM jointly reasons about pose and shape.The paper reports significant improvement over pose-only methods.
- PF-LRM: PF-LRM predicts per-view coarse point clouds from image patch tokens and solves camera poses from 3D–2D correspondences with differentiable PnP.This contrasts with frame-to-frame 3D–3D scene-flow correspondence and suits sparse views with little overlap.
- PF-LRM: PF-LRM uses single-stage inference to recover camera poses and 3D NeRF reconstructions simultaneously.This differs from prior two-stage pipelines that first infer coarse poses and shapes before refinement.
3 METHOD
PF-LRM jointly reconstructs a triplane NeRF and estimates relative camera poses from unposed images using a single-stream transformer. It predicts per-patch 3D geometry and opacity, then applies differentiable weighted PnP alongside rendering and consistency losses.
- Single-stream transformer: PF-LRM tokenizes input images and a triplane NeRF, concatenates their tokens, and processes them with self-attention in a single-stream transformer.The output triplane tokens support volumetric NeRF rendering, while image patch tokens support pose prediction.
- Per-patch geometry prediction: The transformer predicts a 3D point, foreground opacity, and confidence weight for each image patch token.These predictions provide per-view coarse geometry and determine each correspondence’s contribution to pose estimation.
- Online geometry supervision: During training, NeRF-derived ray geometry supervises per-patch point estimates without direct 3D ground-truth supervision.Online distillation from the learned NeRF stabilizes the differentiable PnP loss; without it, training tends to diverge.
- Differentiable PnP pose estimation: Weighted PnP estimates each relative camera pose by minimizing reprojection error between predicted 3D points and 2D patch centers.Predicted opacity suppresses non-informative background points, while confidence weights modulate correspondence contributions.
- Training objective: A robust differentiable PnP loss regularizes the non-convex pose solution, and the final objective combines rendering, geometry, opacity, and pose losses.The method uses the EPro-PnP loss to improve pose prediction and combines losses from NeRF rendering, coarse geometry, opacity, and pose estimation.
4 EXPERIMENTS
PF-LRM is evaluated for cross-dataset pose estimation, reconstruction quality, robustness, ablations, and downstream text/image-to-3D use on unseen and sparse-view settings. It consistently outperforms baselines, while experiments show benefits from its joint pose prediction, large model capacity, and scalable training design.
- Experimental setup: The evaluation uses posed renderings from Objaverse and real captures from MVImgNet for training, with unseen OmniObject3D, GSO, ABO, CO3D, and DTU datasets for cross-dataset testing.The model is trained only with multi-view posed images and is compared with FORGE, RelPose++, and HLoc on pose estimation, plus FORGE on reconstruction quality.
- Pose prediction quality: PF-LRM achieves state-of-the-art pose estimation across unseen datasets, consistently outperforming baselines by a large margin across all reported metrics.On the explicitly sparsified OmniObject3D, GSO, and ABO datasets, rotation error is reduced by 14.6x versus FORGE, 15.3x versus HLoc, and 14.7x versus RelPose++.
- Pose prediction quality: HLoc fails on more than 97% of the very sparse inputs, highlighting the difficulty of feature matching when views have little overlap.The comparison uses sparse inputs from OmniObject3D, GSO, and ABO, where feature matching is especially challenging.
- Reconstruction quality: PF-LRM achieves an average PSNR of 24.8 on OmniObject3D, GSO, and ABO, compared with 13.4 for FORGE.The evaluation also checks whether predicted NeRFs rendered at predicted poses match the input unposed images, testing joint pose and reconstruction consistency.
- Robustness tests: Performance degrades acceptably as the number of input views decreases, while the model supports variable numbers of views without retraining or fine-tuning.The model is trained on four views but is tested with variable numbers of input images on unseen GSO data.
- Ablation studies: The large model outperforms the smaller model in pose accuracy and NeRF reconstruction quality, and removing pose prediction further worsens texture quality.Ablations show sharper textures for joint pose and reconstruction, while removing NeRF prediction makes differentiable PnP training unstable and prone to divergence.
- Downstream applications: PF-LRM can support downstream text-to-3D pipelines by reconstructing a NeRF from 2–4 unposed images after image-conditioned generators produce additional views.This enables a two-stage feed-forward pipeline combining generated multi-view images with PF-LRM reconstruction.
5 CONCLUSION
PF-LRM jointly estimates camera parameters and 3D NeRF shapes with a transformer, outperforming baselines in pose prediction and reconstruction quality. The paper also identifies limitations involving backgrounds, view-dependent appearance, resolution, known intrinsics, and compute-related concerns.
- PF-LRM uses transformer self-attention for communication between triplane and image patch tokens, jointly estimating camera parameters and NeRF-based 3D shapes.The model is trained on posed multi-view data and supports downstream text/image-to-3D generation.
- The model outperforms baseline methods by a large margin in pose prediction accuracy and reconstruction quality.
- The method ignores background cues such as vanishing points and casting shadows when predicting camera poses.Future work is proposed to handle backgrounds with spatial warpings.
- PF-LRM does not model view-dependent effects, limits triplane resolution, and currently assumes known camera intrinsics.The authors identify these as directions for future improvements in appearance, detail, and camera modeling.
- The model raises compute and environmental concerns because its training cost is significantly larger than that of previous 3D reconstruction models.The ethics statement also notes potential copyright concerns when reconstructing shapes from images.
A.1 VISUAL COMPARISONS OF PREDICTED CAMERA POSES
Visual comparisons show that PF-LRM’s predicted camera poses align consistently with ground truth, whereas baseline predictions can deviate substantially. The section also reports category-level pose comparisons against RelPose++ on CO3D.
- PF-LRM’s predicted camera poses consistently align closely with ground truth, unlike FORGE and RelPose++, whose predictions can deviate substantially or fall on the opposite side.
- The CO3D comparison evaluates mean pose errors and rotation accuracy@15° across 10 test categories against RelPose++.
A.2 ADDITIONAL EXPERIMENTS
Additional experiments examine robustness to lighting, qualitative generalization across unseen datasets, and alternative pose predictors. PF-LRM remains consistent under varied illumination, while differentiable PnP performs better than regression-based alternatives.
- Robustness to novel environment lights: PF-LRM shows consistent results under different environment lights on 100 GSO object samples.The model was trained with uniformly lit synthesized multi-view images but was evaluated under novel lighting conditions.
- Ablations of pose prediction methods: Differentiable PnP outperforms MLP-based pose prediction alternatives in the ablation study.The regression-based alternatives are more vulnerable to pose-prediction local minima than the EPro-PnP solver.
- Visual pose comparisons: Predicted poses align more closely with ground truth than those from FORGE and RelPose++.
- Qualitative comparisons: The lighting comparison includes input images under novel and uniform lighting, rendered predictions, novel ground-truth images, and predicted geometry.
- Cross-dataset generalization: PF-LRM qualitatively generalizes to unseen OmniObject3D, GSO, ABO, CO3D, and DTU datasets.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
PF-LRM conditions transformer features on image encodings and camera intrinsics, then processes image and triplane tokens jointly. Its implementation includes independent triplane querying for NeRF rendering and multiple model configurations and memory-saving techniques.
- Feature modulation: Camera intrinsics and view encodings are mapped to modulation features that provide scale and bias for adaptive layer normalization in each transformer block.
- Transformer and rendering: Image tokens are concatenated with learnable triplane positional embeddings and processed by a single-stream transformer.During rendering, the three planes are queried independently and their features are concatenated for the NeRF MLP.
- Model configurations: PF-LRM has small and large configurations with 24 and 36 self-attention layers, respectively.
- Training efficiency: Training uses BFloat16 mixed precision, deferred NeRF back-propagation, gradient checkpointing, and FlashAttention V2 to reduce memory use and training time.
A.4 ADDITIONAL RESULTS
On CO3D, PF-LRM outperforms RelPose++ in most categories despite not being trained on CO3D, while noisy background-removal masks limit its performance.
- PF-LRM outperforms RelPose++ on 8 of 10 held-out CO3D categories despite not using CO3D training data.
- Noisy CO3D masks negatively affect PF-LRM because the model currently handles images without backgrounds.
- CO3D evaluation uses dataset-provided masks to remove backgrounds before testing PF-LRM.
- Extending PF-LRM to images with backgrounds is identified as a future direction to reduce the impact of 2D mask errors.
A.5 ADDITIONAL CROSS-DATASET EVALUATIONS
PF-LRM generalizes to GSO renderings produced by FORGE despite their darker appearance, yielding sharper renderings and more accurate camera estimates than FORGE.
- The pose ablation compares geometric point-cloud prediction followed by differentiable PnP against MLP-based pose prediction methods on GSO.
- PF-LRM generalizes well to GSO renderings produced by FORGE despite their darker appearance than the training images.
- PF-LRM produces sharper renderings than FORGE with and without per-scene optimization-based refinement, as indicated by higher SSIM.
- PF-LRM estimates camera poses more accurately than FORGE on the GSO evaluation.
A.6 SCALING UP TRAINING OF RELPOSE++
When RelPose++ is retrained on Objaverse, it improves on unseen datasets, but PF-LRM remains the comparison model in the cross-dataset evaluation.
- Retrained RelPose++ improves over its CO3D-pretrained version on the unseen OmniObject3D, GSO, and ABO test sets.
- The comparison retrains RelPose++ and evaluates it against PF-LRM models trained on the same Objaverse renderings.
- Table 11 compares author-provided and Objaverse-retrained RelPose++ checkpoints with PF-LRM on GSO, ABO, and OmniObject3D.