PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldMirror: Universal 3D World Reconstruction with Any-Prior Prompting

Yifan Liu, Zhiyuan Min, Zhenwei Wang, Junta Wu, Tengfei Wang, Yixuan Yuan, Yawei Luo, Chunchao Guo

arXiv:2510.10726v2cs.CV

TL;DR

WorldMirror addresses the lack of a unified 3D geometry model that can use diverse priors and perform multiple prediction tasks. It tokenizes heterogeneous inputs and uses unified spatial prediction with coordinated training, achieving state-of-the-art results across diverse benchmarks. The approach also reports universal gains from prior injection, while retaining feed-forward inference.

  • Problem

    Existing 3D methods typically use RGB-only inputs or target isolated tasks, lacking a unified model for flexible priors and comprehensive geometric outputs.

  • Method

    WorldMirror uses Multi-modal Tokenization and Unified Spatial Prediction to integrate images, intrinsics, poses, and depth while jointly predicting multiple 3D representations.

  • Results

    WorldMirror achieves state-of-the-art performance across point maps, camera poses, surface normals, and novel view synthesis, with prior injection universally boosting predictions.

  • Takeaways & Limitations

    Unifying input flexibility and multi-task prediction provides a prior-aware architecture for versatile 3D geometric prediction.

  • Takeaways & Limitations

    Performance is suboptimal on dynamic scenes and autonomous-driving environments, and the implementation supports 300–700-pixel inputs but not thousands of views.

Abstract

from arXiv · show

We present WorldMirror, a unified feed-forward model for comprehensive 3D geometric prediction tasks. Unlike existing methods constrained to image-only inputs or customized for a specific task, our framework flexibly integrates diverse geometric priors, including camera poses, intrinsics, and depth maps, while simultaneously generating multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussians. Remarkably, prior injection yields universal gains across all tasks, suggesting that input flexibility and multi-task prediction are mutually reinforcing. WorldMirror achieves state-of-the-art performance across diverse benchmarks from camera, point map, depth, and surface normal estimation to novel view synthesis, while maintaining the efficiency of feed-forward inference.

1. Introduction

WorldMirror addresses the fragmentation of 3D geometry models by combining flexible prior-conditioned inputs with comprehensive multi-task prediction in one feed-forward framework. It reports state-of-the-art performance across diverse geometric tasks and benchmarks.

  • Existing 3D foundation models typically assume RGB-only inputs and specialize in isolated tasks such as depth, point maps, or camera pose prediction.
  • WorldMirror combines flexible input conditioning with comprehensive multi-task 3D prediction in a single end-to-end model.It accepts images, intrinsics, poses, and depth maps while predicting multiple geometric representations.
  • Multi-modal Tokenization encodes images and available geometric modalities as unified tokens, enabling prior integration without architectural modifications.
  • Unified Spatial Prediction coordinates camera, depth, point-map, normal, and novel-view-synthesis tasks through unified decoder heads and curriculum learning.
  • Extensive experiments show state-of-the-art performance across point-map, camera-pose, surface-normal, and novel-view-synthesis benchmarks.The model surpasses VGGT and π3 for point maps and camera estimation, StableNormal and GeoWizard for normals, and AnySplat for novel view synthesis.

2. Related Works

Related work has advanced feed-forward reconstruction, geometric-prior guidance, and multi-task learning separately. WorldMirror brings these directions together by combining heterogeneous inputs with multiple 3D outputs in one feed-forward pass.

  • Feed-Forward 3D Reconstruction: Feed-forward reconstruction models replace costly SfM/MVS optimization with learned prediction and increasingly support scalable, multi-task, and large-scale reconstruction.
  • Feed-Forward 3D Reconstruction: WorldMirror unifies camera poses, depth, normals, point maps, and 3D Gaussians in a single feed-forward pass.
  • 3D Prior Guidance: Prior-guidance methods use intrinsics, camera trajectories, multi-modal cues, additive priors, or per-layer adapters, but explore heterogeneous-prior integration through different designs.
  • Multi-task Learning: 3D multi-task learning faces conflicts between rendering-oriented 3DGS heads and geometry-oriented heads, motivating curriculum learning that trains geometry before the GS head.
  • 3D Prior Guidance: Existing methods separately emphasize prior-conditioned reconstruction or multi-task prediction, whereas WorldMirror targets both capabilities together.
  • Generalizable Novel View Synthesis: Generalizable novel-view synthesis reduces per-scene optimization, but many methods rely on known camera parameters, fixed view counts, or have other limitations.

3. Method

WorldMirror tokenizes images and optional geometric priors into a unified sequence, then uses shared transformer features and task-specific decoders for comprehensive 3D prediction. Decoupled modeling and sequential training address conflicts between geometry and appearance learning.

  • Method Overview: WorldMirror encodes images, intrinsics, poses, and depth maps as tokens and predicts point maps, cameras, depth, normals, and 3D Gaussians.
  • Modality-Specific Tokenization: Camera poses are normalized using the scene centroid and maximum camera-to-centroid distance before projection into pose tokens.
  • Modality-Specific Tokenization: Intrinsics are represented by normalized focal lengths and principal points, then projected into intrinsic tokens.
  • Modality-Specific Tokenization: Depth maps are patch-embedded into spatially aligned tokens and added directly to visual tokens rather than concatenated.This preserves spatial structure and avoids doubling the token count and quadratic attention cost.
  • Flexible Token Merging: Independent random dropping of camera and depth tokens during training enables flexible control over input modalities at inference time.
  • Training Strategy: Decoupled sequential training handles geometry before appearance because jointly optimizing all tasks makes geometry and appearance difficult to disentangle.
  • Geometry Modeling: A transformer backbone with DPT and MLP decoders produces dense point maps, depths, normals, and camera parameters from multi-view features.Surface normals are L2-normalized, and pseudo normals from ground-truth depth supplement scarce annotations.
  • Appearance Modeling: The 3DGS decoder predicts positions, colors, opacity, scale, and rotation, using predicted depths and ground-truth cameras for Gaussian construction and rendering.

4. Model Training

WorldMirror is trained end-to-end with a composite loss supervising multiple geometric outputs and 3D Gaussian prediction. The paper reports qualitative comparisons for surface normals and multi-resolution novel view synthesis.

  • Composite Training Objective: WorldMirror minimizes a composite loss combining point-map, depth, camera, surface-normal, and 3D Gaussian supervision.The Gaussian term is trained through rendering losses.
  • Surface Normal Estimation: Table 3 compares surface-normal estimation on ScanNet, NYUv2, and iBims-1 against regression-based and diffusion-based approaches.
  • Novel View Synthesis: Table 4 evaluates multi-resolution novel view synthesis on DL3DV, marking pose-free optimization for fair comparison with non-pose-free baselines.
  • Novel View Synthesis: Figure 4 compares WorldMirror with FLARE and AnySplat on RealEstate10K and DL3DV across sparse-view and dense-view settings.The first four columns are sparse-view, while the latter three are dense-view.
  • Novel View Synthesis: The qualitative comparison reports higher appearance fidelity and geometric perception than the baselines in both viewing regimes.

5. Experiments

WorldMirror is evaluated across reconstruction, camera, surface-normal, and novel-view-synthesis tasks, including prior-guidance and architectural ablations. Results report strong performance across diverse benchmarks and consistent benefits from priors and the unified design.

  • Evaluation Scope: WorldMirror is evaluated on point-map reconstruction, camera-pose estimation, surface-normal estimation, and novel view synthesis.The experiments also assess prior guidance, embedding strategies, and novel-view-synthesis design choices.
  • Point Map Reconstruction: 10.4% and 17.8% accuracy gains over the stated baselines are reported on 7-Scenes and DTU without priors.Adding all priors further improves results, with 58.1% and 53.1% accuracy gains over the no-prior baseline on 7-Scenes and NRGBD.
  • Camera Pose Estimation: WorldMirror achieves superior zero-shot camera-pose performance on RealEstate10K and TUM-dynamics while remaining competitive on Sintel.
  • Surface Normal Estimation: Multi-task learning with shared representations substantially improves surface-normal estimation over existing specialized approaches.Evaluation covers iBims-1, NYUv2, and ScanNet using angular-error and threshold metrics.
  • Novel View Synthesis: WorldMirror consistently outperforms DepthSplat at 8, 24, and 64 DL3DV views, with gains increasing as view count grows.Dynamic-resolution training supports generalization across varying resolutions, while the authors acknowledge broader training data as an advantage.
  • Prior Guidance: Combining camera poses, calibrated intrinsics, and depth priors yields further reconstruction gains through complementary geometric cues.
  • Prior-Guided Comparisons: WorldMirror consistently outperforms Pow3R and MapAnything across most tested prior conditions.The comparison extends Pow3R to multi-view scenarios using Procrustes alignment for camera-pose estimation.
  • Prior Embedding Ablation: Single-token prior embeddings provide superior performance and high efficiency in the reported ablation.

6. Conclusion

WorldMirror unifies flexible prior conditioning with comprehensive multi-task 3D prediction in a single feed-forward framework. Its experiments support prior-aware unified architectures as a promising direction for versatile 3D understanding.

  • Conclusion: WorldMirror integrates images, camera intrinsics, poses, and depth maps as tokens without architectural modifications.Its Unified Spatial Prediction covers camera estimation through novel view synthesis, and prior injection universally boosts predictions.
  • Conclusion: Experiments report state-of-the-art performance across point maps, camera poses, surface normals, and novel view synthesis.
  • Conclusion: The authors identify unified, prior-aware architectures as a promising direction for versatile 3D understanding.

Impact Statement

WorldMirror is presented as enabling efficient 3D scene reconstruction across multiple applications, while the paper acknowledges privacy, environmental misrepresentation, and bias concerns.

  • WorldMirror enables efficient 3D scene reconstruction across multiple applications.
  • The authors identify privacy implications, potential misrepresentation of environments, and algorithmic biases as concerns surrounding implementation.
  • The training objective integrates supervision for all prediction tasks through a composite loss function.
  • Novel-view robustness training selects context and novel-view splits using pixel overlap between depth maps and camera parameters.
  • A dedicated depth supervision loss constrains 3D Gaussian splat locations using ground-truth and GS-head-predicted depth maps.

A.2. Training Settings

The training setup uses dynamic resolutions and batch sizing, a two-phase optimization process, weighted composite losses, and heterogeneous data spanning 15 datasets.

  • The model trains for 100 epochs with multi-modal prior prompting, then fine-tunes for 50 epochs with a Gaussian head.
  • Dynamic image resolutions maintain total pixel counts between 100,000 and 250,000 while sampling aspect ratios from 0.5 to 2.0.
  • The reported benchmark settings include monocular and video depth estimation on NYUv2, Sintel, and KITTI, plus NVS with 3DGS optimization on three datasets.
  • The composite loss weights camera supervision at λcam = 5.0, while point, depth, normal, and 3DGS terms each use weight 1.0.
  • Training data comprises a diverse collection of 15 datasets covering varied scene types and capture conditions.

B.1. Monocular and video depth benchmark

WorldMirror performs competitively across depth estimation and novel-view synthesis evaluations, with strong NVS generalization but a reported KITTI gap attributed to training-data coverage.

  • Monocular and video depth benchmark: WorldMirror matches or exceeds leading monocular metric-depth methods and rivals specialized feed-forward reconstruction frameworks on video sequences.
  • Monocular and video depth benchmark: WorldMirror has a modest performance gap on KITTI relative to π3, attributed to under-represented urban driving environments in training.
  • Novel view synthesis: WorldMirror substantially improves over AnySplat across all reported novel-view-synthesis metrics on both evaluated datasets at 518 × 378 resolution.
  • Novel view synthesis: On MatrixCity, WorldMirror generalizes beyond the 24-view training maximum and outperforms feed-forward and optimization-based approaches without post-processing.
  • Novel view synthesis: In two-view NVS, WorldMirror is comparable to NoPoSplat across most metrics and substantially outperforms AnySplat despite differing training configurations.
  • Prior robustness: The model exhibits graceful degradation under noisy priors on the 7-Scenes and DTU datasets.
  • Prior ablation: A single modality prior improves performance across corresponding and other tasks, as illustrated by metrics including inlier ratio, AUC@5, and average focal error.
  • Training ablation: Decoupled training consistently outperforms joint training across nearly all benchmarks under matched 32,500-step, 32-GPU conditions.

C.2. Sensitivity Analysis on Prior Dropout Probability

The prior-dropout sweep evaluates robustness to withheld depth and normal priors under both no-prior and all-prior inference conditions.

  • Prior dropout probability p controls how often depth and normal priors are withheld during training.
  • The sweep tests p ∈{0.0, 0.3, 0.5, 0.7, 1.0} under no-prior and all-prior inference regimes.
  • The evaluation reports performance under both inference conditions, with lower values preferred for all accuracy metrics.

D. Robustness to Noisy or Low-Quality Priors

WorldMirror remains useful with noisy camera, intrinsic, and depth priors, while normalized depth improves relative-scale reconstruction. Its dense prediction tasks scale to many views, although Gaussian prediction requires substantially more memory and training is resource-intensive.

  • Noise robustness: Controlled noise experiments simulate sensor inaccuracies and calibration errors across camera pose, camera intrinsic, and depth priors.The evaluation uses 7-Scenes and DTU, comparing reconstruction quality against a no-prior baseline.
  • Noise robustness: WorldMirror degrades gracefully as prior noise increases, with camera-pose priors remaining better than the no-prior baseline up to 20° rotation error.Moderate noise still provides meaningful guidance, whereas depth priors are more sensitive because they directly guide geometry.
  • Depth-prior design: Addition achieves superior accuracy with optimal efficiency, while concatenation increases computational cost by 52.6% with comparable performance.Direct addition preserves pixel-wise spatial correspondence and avoids expensive attention operations.
  • Depth-prior design: Normalized depth provides more stable geometric priors for relative-scale reconstruction across scenes with different absolute scales.Normalization maps depth values into a unified [0,1] range and improves training stability.
  • Resource requirements: Most dense prediction tasks scale from approximately 5GB for one view to approximately 23GB for 256 views, while 3D Gaussian Splatting reaches 60.5GB.Gaussian prediction uses additional convolutional layers to decode high-dimensional Gaussian attributes.
  • Resource requirements: Dense prediction tasks support up to 1024 input views, whereas 3D Gaussian Splatting is limited to 360 views because of its higher memory footprint.The reported measurements use a single H20 GPU at 518 × 378 resolution.
  • Resource requirements: WorldMirror’s two-stage training takes approximately 70 hours on 32 NVIDIA H20 GPUs, including 42 hours for dense heads and 28 hours for the Gaussian head.Gradient checkpointing, Flash Attention v3, and BF16 mixed precision are used for efficiency.

F.4. Inference Speed and Practical Usability.

WorldMirror’s prior injection adds negligible latency in the Geo model, while the full model completes five tasks in one forward pass with approximately 13% overhead at 64 views. Its outputs support plausible Gaussian rendering and cleaner surface reconstruction, but limitations remain for dynamic scenes, high view counts, and consumer hardware.

  • Inference speed: WorldMirror’s Geo model matches VGGT’s latency, indicating negligible computational cost from prior injection.Median inference times are measured on a single NVIDIA H20 GPU at 518 × 378 resolution.
  • Inference speed: The full model introduces approximately 13% overhead at 64 views while completing all five tasks in a single forward pass.At 64 views, Geo uses 9.96 GB and Full uses 17.82 GB, and both fit on a single GPU.
  • Limitations: WorldMirror performs suboptimally on dynamic scenes and autonomous-driving environments because such data are under-represented in its training distribution.The current implementation also supports 300–700-pixel inputs and cannot effectively handle thousands of views, especially on consumer GPUs.
  • 3D Gaussian and rendering results: WorldMirror’s feed-forward Gaussian outputs produce plausible geometric structures and high-quality novel-view renderings for AI-generated videos and real multi-view images.The results are described as consistent across these diverse input scenarios.
  • Point-map reconstruction: WorldMirror produces more consistent point-map reconstructions on benchmark and in-the-wild images, including sparse viewpoints and challenging viewing conditions.The comparisons cover 7-Scenes, NRGBD, DTU, video-generation outputs, and real-world multi-view images.
  • Surface reconstruction: Predicted smooth normal maps enable cleaner surfaces with sharp details through Poisson surface reconstruction.This approach replaces traditional geometric normal estimation from point clouds.
Loading 2510.10726v2…