PaperlayerView exact revision on arXiv

Source-linked AI summary

$π^3$: Permutation-Equivariant Visual Geometry Learning

Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, Tong He

arXiv:2507.13347v3cs.CV

TL;DR

Visual geometry reconstruction commonly relies on a fixed reference view, creating an inductive bias and sensitivity to reference selection. π3 removes that dependence with a fully permutation-equivariant, reference-free architecture that predicts affine-invariant poses and scale-invariant local pointmaps. It achieves state-of-the-art performance across camera pose, depth, and dense reconstruction tasks, with representative Sintel improvements in pose and video depth.

  • Problem

    Fixed reference views remain a common design choice in visual geometry reconstruction and introduce an inductive bias that limits robustness and performance.

  • Method

    π3 uses a fully permutation-equivariant architecture to predict affine-invariant camera poses and scale-invariant local pointmaps without a designated reference view.

  • Results

    π3 achieves state-of-the-art performance across camera pose estimation, depth estimation, and pointmap reconstruction, including Sintel pose ATE of 0.074 versus VGGT’s 0.167.

  • Takeaways & Limitations

    Reference-free systems can provide stable, versatile visual geometry reconstruction across varied inputs and static or dynamic scenes.

  • Takeaways & Limitations

    π3 cannot handle transparent objects, lacks diffusion-based fine-grained detail, and may produce grid-like artifacts from pixel-shuffling upsampling.

Abstract

from arXiv · show

We introduce $π^3$, a feed-forward neural network that offers a novel approach to visual geometry reconstruction, breaking the reliance on a conventional fixed reference view. Previous methods often anchor their reconstructions to a designated viewpoint, an inductive bias that can lead to instability and failures if the reference is suboptimal. In contrast, $π^3$ employs a fully permutation-equivariant architecture to predict affine-invariant camera poses and scale-invariant local point maps without any reference frames. This design not only makes our model inherently robust to input ordering, but also leads to higher accuracy and performance. These advantages enable our simple and bias-free approach to achieve state-of-the-art performance on a wide range of tasks, including camera pose estimation, monocular/video depth estimation, and dense point map reconstruction. Code and models are available at https://github.com/yyfz/Pi3.

1 INTRODUCTION

π3 removes the fixed-reference-view bias through a fully permutation-equivariant architecture, improving robustness, accuracy, and performance across visual geometry tasks.

  • Fixed reference views introduce an unnecessary inductive bias that constrains the robustness and performance of visual geometry models.
  • π3 eliminates reference-view dependence by predicting affine-invariant camera poses and scale-invariant local pointmaps through purely relative, per-view geometry.
  • π3 accepts single images, video sequences, and unordered image sets from static or dynamic scenes without designating a reference view.
  • π3 maintains stable performance when the reference frame changes, with minimal degradation and low standard deviation compared with previous methods.
  • On Sintel, π3 reduces camera-pose ATE from 0.167 to 0.074 and scale-aligned video-depth absolute relative error from 0.299 to 0.233.
  • π3 establishes state-of-the-art results across camera pose, monocular/video depth, and pointmap reconstruction benchmarks.

2 RELATED WORK

Classical and feed-forward reconstruction methods commonly anchor geometry to a designated reference frame, while π3 departs from that paradigm with reference-free permutation equivariance.

  • 2.1 TRADITIONAL 3D RECONSTRUCTION: Classical SfM and MVS estimate camera poses and dense point clouds through multi-stage pipelines that often use iterative Bundle Adjustment.
  • 2.2 FEED-FORWARD 3D RECONSTRUCTION: Feed-forward models directly regress scene geometry in a single pass, but early pairwise approaches define point clouds in the first camera’s coordinates.
  • 2.2 FEED-FORWARD 3D RECONSTRUCTION: Scaling pairwise feed-forward reconstruction to larger scenes can require global alignment, which may be time-consuming and unstable.
  • 2.2 FEED-FORWARD 3D RECONSTRUCTION: Fast3R removes costly global alignment for thousands of images, while FLARE separates pose prediction from geometry estimation and VGGT uses multi-task learning.
  • 2.2 FEED-FORWARD 3D RECONSTRUCTION: Despite these advances, feed-forward methods retain the inherited paradigm of anchoring predicted 3D structure to a designated reference frame.

3 METHOD

π^3 uses permutation-equivariant processing to predict per-view camera poses, local point maps, and confidence maps without a designated reference frame. Its training handles scale ambiguity and supervises relative poses, while the learned pose distributions exhibit low-dimensional structure.

  • 3.1 PERMUTATION-EQUIVARIANT ARCHITECTURE: Permutation-equivariant processing preserves one-to-one correspondence between reordered input images and their predicted poses, point maps, and confidence maps.The network omits order-dependent components and is designed so outputs are identically permuted when inputs are permuted.
  • 3.2 SCALE-INVARIANT LOCAL GEOMETRY: π^3 predicts pixel-aligned 3D point maps in each image’s local camera coordinate system, with one consistent unknown scale across all views.Training aligns all predicted point maps to ground truth using a single optimal scale factor.
  • 3.2 SCALE-INVARIANT LOCAL GEOMETRY: The point-map objective combines scale-aligned reconstruction, locally smooth surface normals, and confidence supervision based on reconstruction error.Normals are supervised through angular agreement, while confidence targets indicate whether scaled L1 error falls below a threshold.
  • 3.3 AFFINE-INVARIANT CAMERA POSE: Relative-pose supervision resolves global-frame ambiguity by combining rotation and scaled translation losses over ordered view pairs.A shared scale factor rectifies predicted camera translations, and Huber loss is used for translation robustness to outliers.
  • 3.3 AFFINE-INVARIANT CAMERA POSE: The model’s predicted camera poses have variance concentrated along fewer principal components than VGGT, confirming a low-dimensional output structure.Figure 4 compares the predicted pose distributions, while eigenvalue analysis provides the quantitative confirmation.

4 EXPERIMENTS

Experiments evaluate π3 across camera pose, point map, video depth, monocular depth, robustness, and ablation settings. The results show state-of-the-art or comparable performance, strong generalization, high efficiency, and substantially greater invariance to input ordering and reference-view selection.

  • Overall results: π3 achieves state-of-the-art or comparable performance across camera pose, point map, video depth, and monocular depth estimation tasks.Evaluations cover multiple synthetic and real-world datasets, sparse and dense views, and object-level and scene-level reconstruction.
  • 4.1 Camera pose estimation: π3 sets a new camera-pose benchmark on Sintel and RealEstate10K while achieving competitive SOTA results on TUM-dynamics, Co3Dv2, and ScanNet.These results indicate strong zero-shot generalization alongside performance on familiar data distributions.
  • 4.3 Depth estimation: 57.4 FPS on KITTI exceeds VGGT’s 43.2 FPS and Aether’s 6.14 FPS for video depth estimation despite π3’s smaller model size.FPS is evaluated on KITTI using one A800 GPU.
  • 4.4 Robustness evaluation: π3’s mean accuracy standard deviation on DTU is 0.003 versus VGGT’s 0.033, while ETH3D shows effectively zero variance across input orderings.Lower standard deviation indicates greater robustness to input sequence permutations.
  • 4.5 Ablation study: Affine-invariant camera pose modeling consistently improves final performance and makes the model permutation-equivariant, while scale-invariant pointmaps help more substantially on outdoor data.Scale-invariant pointmap modeling yields limited gains on indoor 7-Scenes and NRGBD but larger gains outdoors.

5 CONCLUSION

π3 removes the fixed reference view through a fully permutation-equivariant architecture. The resulting reference-free system is robust to input ordering and achieves state-of-the-art performance across camera pose, depth, and dense reconstruction tasks.

  • 5 CONCLUSION: π3 eliminates fixed-reference-view dependence and uses permutation equivariance to improve robustness and accuracy.The conclusion presents this as a new paradigm for visual geometry reconstruction.
  • 5 CONCLUSION: π3 achieves state-of-the-art performance across camera pose estimation, depth estimation, and dense reconstruction.The conclusion characterizes reference-free systems as viable for stable and versatile 3D vision models.

A.1 ARCHITECTURE DETAILS

π3 uses a lightweight transformer-based architecture with separate decoders for camera poses, local point maps, and confidence scores. Task-specific output heads produce the corresponding predictions.

  • A.1 ARCHITECTURE DETAILS: π3 uses a lightweight 5-layer transformer whose self-attention operates exclusively on each image’s features.Its encoder and alternating attention modules follow VGGT, but the alternating attention module uses 36 layers instead of 48.
  • A.1 ARCHITECTURE DETAILS: Separate, non-weight-sharing decoders predict camera poses, local point maps, and confidence scores.The decoders share an architecture but have distinct weights.
  • A.1 ARCHITECTURE DETAILS: Point-map and confidence heads use an MLP followed by pixel shuffle, while the camera-pose head uses MLPs, average pooling, and SVD-based rotation orthogonalization.Camera rotation is initially represented in 9D and converted to a 3×3 rotation matrix.

A.2 TRAINING DETAILS

π3 is trained in two stages using low-resolution and variable-resolution images, with dynamic batch sizing and initialization from pretrained VGGT components. The training setup uses frozen encoder weights and multi-GPU training.

  • A.2 TRAINING DETAILS: π3 first trains at 224 × 224 resolution, then fine-tunes on random resolutions with 100,000–255,000 total pixels and aspect ratios from 0.5 to 2.0.Each stage runs for 80 epochs with 800 iterations per epoch.
  • A.2 TRAINING DETAILS: The model initializes its encoder and alternating attention weights from pretrained VGGT and keeps the encoder frozen during training.The training procedure is not conducted entirely from scratch.

A.3 DISCUSSION FOR PREDICTED POSE DISTRIBUTION

Figure 6 visualizes predicted camera poses using spatial coordinates for translation and RGB colors for rotation, revealing the geometry of the learned representations.

  • Translation components are plotted as spatial coordinates (x, y, z), while rotation components are represented through RGB colors.Rotation matrices are converted to normalized axis-angle vectors before mapping their components to the red, green, and blue channels.

A.4 COMPARISON WITH VGGT

The comparison with VGGT addresses optimization instability in relative pose supervision by adding a global pointmap proxy task while retaining a permutation-equivariant final model. With this stabilization, π3 outperforms VGGT on ETH3D and NRGB benchmarks and produces coherent in-the-wild reconstructions.

  • Optimization challenge: Relative pose supervision creates a cold-start problem because π3 generates highly coupled N × N constraints that are unstable from random initialization.Reference-anchored methods avoid this specific optimization difficulty by providing a fixed frame for supervision.
  • Optimization challenge: An auxiliary global pointmap head uses a reference frame only as a proxy task to decouple geometry learning and stabilize optimization.The final model remains fully permutation-equivariant despite using cross-attention to the reference view in this auxiliary head.
  • Experimental comparison: The adapted π3 model and VGGT are trained under identical multi-task conditions, including the same data, resolution, and 80-epoch schedule.Training is from scratch except for DINOv2 encoders, using 800 steps per epoch at 224 × 224 resolution.
  • Experimental comparison: Once stabilized by the global proxy, π3 significantly outperforms the VGGT baseline on ETH3D and NRGB benchmarks.The main experiments nevertheless use VGGT initialization for computational efficiency and to leverage pretrained data priors.
  • Qualitative comparison: π3 produces more coherent and complete 3D structures than other feed-forward approaches on challenging in-the-wild dynamic and complex static scenes.

A.5 CAMERA POSE EVALUATION METRICS

The paper evaluates camera poses with angular and trajectory-based metrics across synthetic and real-world datasets, alongside additional point-map and monocular-depth evaluations. It also examines tighter angular thresholds, compares monocular depth with Depth Anything V2, and reports limitations affecting reconstruction quality.

  • Angular accuracy metrics: Relative Rotation Accuracy and Relative Translation Accuracy measure angular errors from all image pairs sampled within RealEstate10K and Co3Dv2 sequences.Each sequence contributes 10 randomly sampled images, whose possible pairs are evaluated for relative rotation and translation errors.
  • Distance error metrics: ATE, RPE-t, and RPE-r evaluate predicted camera trajectories on Sintel, TUM-dynamics, and ScanNet after Sim(3) alignment with ground truth.
  • Angular accuracy metrics: Tighter thresholds of 1°, 3°, 5°, 10°, and 15° supplement the relaxed 30° evaluation for RRA, RTA, and AUC on RealEstate10K.
  • Additional evaluation: Chamfer Distance is additionally reported for point-map estimation as the mean of Accuracy and Completion terms.
  • Additional evaluation: π3 achieves comparable monocular-depth performance to specialized Depth Anything V2 despite being designed for generalist multi-view reconstruction.Evaluation follows the CUT3R protocol at 518 input resolution.
  • Limitations: The model cannot handle transparent objects, lacks diffusion-level fine-grained detail, and may produce grid-like artifacts from MLP pixel-shuffling upsampling.Artifacts are especially noticeable in regions with high reconstruction uncertainty.
Loading 2507.13347v3…