Source-linked AI summary
ViPE: Video Pose Engine for 3D Geometric Perception
Jiahui Huang, Qunjie Zhou, Hesam Rabeti, Aleksandr Korovko, Huan Ling, Xuanchi Ren, Tianchang Shen, Jun Gao, Dmitry Slepichev, Chen-Hsuan Lin, Jiawei Ren, Kevin Xie, Joydeep Biswas, Laura Leal-Taixe, Sanja Fidler
TL;DR
Existing datasets and methods do not adequately provide consistent, precise geometric annotations for diverse in-the-wild videos, although such geometry supports spatial AI applications. ViPE combines classical and learned components to estimate camera parameters and dense depth, then releases large annotated video datasets. It is designed for efficient, robust processing and has supported downstream world-generation and 3D reconstruction applications.
Problem
Existing datasets often lack the scale, scene diversity, and high-quality geometric and pose annotations needed across vision and robotics tasks.
Method
ViPE combines bundle adjustment with dense optical-flow and sparse keypoint constraints, dynamic-object handling, multiple camera models, and depth alignment for video pose and depth estimation.
Results
ViPE provides accurate camera poses, intrinsics, and dense depth maps, runs at 3-5FPS on a single GPU, and accompanies a release containing 96 million annotated frames.
Takeaways & Limitations
The released datasets have supported training-data annotation, conditional buffers for world generation, and feed-forward 3D Gaussian Splatting reconstruction.
Takeaways & Limitations
In rare extreme cases with few or no pixels covered by projected depth, the usual affine alignment is not constrained and ViPE directly assigns the projected depth map.
Abstract
from arXiv · showhide
Accurate 3D geometric perception is an important prerequisite for a wide range of spatial AI systems. While state-of-the-art methods depend on large-scale training data, acquiring consistent and precise 3D annotations from in-the-wild videos remains a key challenge. In this work, we introduce ViPE, a handy and versatile video processing engine designed to bridge this gap. ViPE efficiently estimates camera intrinsics, camera motion, and dense, near-metric depth maps from unconstrained raw videos. It is robust to diverse scenarios, including dynamic selfie videos, cinematic shots, or dashcams, and supports various camera models such as pinhole, wide-angle, and 360° panoramas. We have benchmarked ViPE on multiple benchmarks. Notably, it outperforms existing uncalibrated pose estimation baselines by 18%/50% on TUM/KITTI sequences, and runs at 3-5FPS on a single GPU for standard input resolutions. We use ViPE to annotate a large-scale collection of videos. This collection includes around 100K real-world internet videos, 1M high-quality AI-generated videos, and 2K panoramic videos, totaling approximately 96M frames -- all annotated with accurate camera poses and dense depth maps. We open-source ViPE and the annotated dataset with the hope of accelerating the development of spatial AI systems.
1. Introduction
ViPE bridges classical optimization and learned perception to estimate camera geometry and dense metric depth from diverse in-the-wild videos. It also supports large-scale annotation through an efficient pipeline and a released 96-million-frame dataset.
- ViPE combines dense Bundle Adjustment with learned components to estimate camera poses, intrinsics, and dense metric depth from challenging in-the-wild videos.The design targets the scalability and precision of SLAM alongside the robustness of modern learned models.
- 3-5FPS: ViPE typically processes standard-resolution inputs on a single GPU.The measurement uses 640 × 480 inputs on an NVIDIA RTX 5090 GPU.
- ViPE improves efficiency, dynamic-object handling, camera-model support, and camera-estimation accuracy relative to prior work.Compared with MegaSAM, it does not require per-frame optimization and supports a wider variety of camera models.
- 96 million annotated frames: the released collection combines approximately 100K real-world videos, 1M AI-generated videos, and panoramic videos.The components are Dynpose-100K++, Wild-SDG-1M, and Web360.
2. Related Works
Prior systems trade off geometric precision, robustness, scalability, or dense metric geometry on diverse in-the-wild videos. ViPE is positioned as a unified framework for producing large-scale, high-quality video annotations.
- Classical and hybrid methods: Hybrid SLAM systems integrate learned front-ends with classical back-ends, but loose coupling can leave inconsistencies unresolved.
- Dense metric geometry: Metric-scale dense geometry can be supported by monocular metric-depth and video-depth models used as optimization priors and refinement sources.
- Annotated video datasets: Large-scale video datasets need diverse scenes and high-quality pose and geometry annotations, while existing datasets often lack these properties.The paper positions its collection as combining large-scale, diverse real-world content with high-quality annotations.
3. Methodology
ViPE’s methodology combines dense and sparse constraints in bundle adjustment and adds depth alignment to produce consistent per-frame depth maps.
- The methodology is organized around the pipeline, the core Bundle Adjustment formulation, and the final depth alignment stage.
3.1. Overview
ViPE uses a keyframe-based SLAM design for scalable processing of videos of arbitrary lengths. It initializes intrinsics, selects keyframes, tracks locally, and optimizes all current keyframes globally.
- ViPE uses a keyframe-based SLAM system to improve scalability and robustness for videos of arbitrary lengths.Its frontend and backend design primarily follows DROID-SLAM.
- Intrinsics initialization runs GeoCalib on four uniformly sampled video frames before keyframe processing.
- Keyframes are selected when predicted motion from the current frame to the previous keyframe exceeds a predefined threshold.Motion combines weighted dense optical flow and sparse keypoint tracks.
- Frontend tracking builds a small sliding-window graph, while backend optimization solves a full Bundle Adjustment problem over current keyframes.Graph edges reflect temporal proximity or sufficient co-visibility.
3.2. Formulation
ViPE formulates video geometry estimation as a scalable bundle-adjustment problem combining dense and sparse correspondences with depth regularization. It extends this formulation to dynamic scenes, multiple camera models, and multi-camera or panoramic inputs.
- 3.2. Formulation: ViPE jointly optimizes frame poses, camera intrinsics, and low-resolution depth maps for keyframes in a graph.The objective combines dense matching, sparse keypoint matching, and depth regularization, solved with a Gauss-Newton factorized solver.
- 3.2.1. Dense Flow Constraint: Dense optical flow supplies low-resolution correspondence constraints, with h = H/8 and w = W/8 reducing the optimized depth variables.A learned flow network estimates Fᵢⱼ at the depth-map resolution, while confidence and motion weights modulate the constraint.
- 3.2.2. Sparse Point Constraint: Sparse keypoint tracking complements dense flow by recovering fine image details that low-resolution network inference may miss.The sparse constraint uses matched keypoints and bilinear depth interpolation, while bilinear splatting is adopted to avoid numerous depth-map interactions in the Hessian.
- 3.2.3. Depth Regularization: Metric depth priors regularize estimation, reduce scale drift, and provide real-world metric-scale information from single images.The method supports Metric3dv2, UniDepthV2, and UniK3D, and updates depth predictions after intrinsics optimization.
- 3.2.4. Dynamic Object Masking: Semantic masks identify dynamic objects and propagate their segmentation through XMem instead of running segmentation models on every frame.The inverted masks weight dense constraints toward static background regions and reject sparse tracks outside those regions as outliers.
- 3.2.5. Handling Different Camera Models: ViPE supports pinhole and wide-angle cameras by optimizing intrinsic parameters under a radial formulation, including distortion strength α in the unified camera model.For pinhole cameras, focal length is the only estimated scalar; for wide-angle cameras, k = [f, α], with α = 0 reducing to the pinhole model.
- 3.2.5. Handling Different Camera Models: The BA formulation extends to multi-camera rigs by incorporating rig-to-camera transformations and adaptively adding cross-view edges based on co-visibility.For 360° videos, panoramas are projected into six roughly cubical pinhole views covering front, back, left, right, up, and bottom directions.
3.3. Post-processed Dense Depth Alignment
ViPE aligns temporally smooth affine-invariant video depth with bundle-adjustment depth to produce consistent dense maps across frames. It handles incomplete projected depth using coverage checks and infilling, with a fallback for extreme cases.
- State-of-the-art depth networks provide temporally smooth relative depth, while bundle-adjustment depth better aligns with camera poses but can be noisy or incomplete.
- ViPE combines video depth estimation with bundle-adjustment depth to obtain smooth, pose-consistent dense depth maps.The video depth map supplies temporal smoothness, while bundle-adjustment depth supplies alignment with estimated poses.
- A momentum-based update estimates affine transformation parameters before producing the final aligned depth map.The parameters are updated from their previous estimates and current-frame estimates using momentum factor m.
- When projected bundle-adjustment depth lacks sufficient coverage, ViPE infills it with PriorDA conditioned on partial depth observations and the input image.
- In very extreme and rare cases with few or no covered pixels, ViPE directly assigns the projected depth map instead of performing the usual alignment.
4. Evaluation
ViPE is evaluated on standard and unposed videos for camera pose, intrinsics, and depth estimation. It shows competitive accuracy, metric-scale behavior, reliable pose consistency, and robustness to dynamic scenes and accumulated 3D reconstruction.
- Evaluation setup: ViPE is evaluated on camera intrinsics, poses, and depth across standard benchmarks and unposed real-world videos.The evaluation includes TUM RGB-D, outdoor driving datasets, OpenDV, VidBench, SINTEL, and ETH3D.
- Standard benchmarks: ViPE reaches competitive pose and intrinsics performance on indoor and outdoor datasets while efficiently removing movable objects from camera estimation.Pose quality is measured with ATE, RTE, RRE, and focal-length error.
- Standard benchmarks: ViPE’s output pose is roughly aligned with real-world scale, whereas MegaSAM produces pose in an indefinite scale space.The metric depth prediction module supports this scale alignment in the KITTI qualitative comparison.
- Unposed videos: Two ground-truth-free metrics evaluate pose consistency on unposed videos: Shuttle Pose Error compares forward and reversed trajectories, while Sampson Error measures epipolar correspondence distance.The trajectories are normalized to length 1 before rigid alignment, and correspondences are detected by LightGlue.
- Unposed videos: ViPE achieves better shuttle consistency and lower Sampson error than the baselines, indicating more reliable estimated camera poses in real-world scenarios.The proposed focal and pose errors also generally correlate with standard pose errors, with reported Pearson correlation r = 0.614.
- Depth estimation: Across SINTEL and ETH3D, ViPE avoids the frame-to-frame jitter seen in monocular depth methods and is generally robust against multi-layer artifacts in accumulated point clouds.The comparison attributes ViPE’s temporal stability to its video depth model.
- Ablation study: Ablations show that sparse track terms and dynamic masking improve robustness, while the depth estimation module further improves accuracy.These components are evaluated on OpenDV and VidBench, which reflect real-world video conditions.
5. Dataset Release
The release addresses scarce, diverse, large-scale geometric data with three ViPE-annotated datasets spanning real-world, AI-generated, and panoramic videos. Together, they provide varied material for training and evaluating 3D geometric perception models.
- Dataset Release: Three datasets provide ViPE-annotated camera poses and geometric information across diverse video sources and content.They were introduced to address the scarcity of high-quality, diverse, large-scale datasets for unconstrained 3D geometric perception.
- Dataset Release: Dynpose-100K++ re-annotates 99,501 real-world internet videos with 15.7M frames spanning approximately 150 hours.The original dataset provided lower-framerate poses without per-frame geometry, motivating the re-annotation.
- Dataset Release: Wild-SDG-1M contains approximately 1 million AI-generated videos and approximately 78 million annotated frames.The videos were sampled from video diffusion models using curated and balanced text prompts.
- Impact: The datasets span dynamic real-world internet videos, synthetic environments, and specialized panoramic content for downstream development.Their scale and diversity support training and evaluation under varied challenging conditions.
- Qualitative Results: Qualitative examples demonstrate consistent and robust ViPE annotations across highly diverse video types.The examples emphasize conditions where other methods may struggle or produce incomplete estimations.
6. Conclusion
ViPE estimates camera poses, intrinsics, and depth maps using bundle adjustment with dense and sparse constraints plus cross-frame depth alignment. Its benchmarks show superior performance, and its outputs support downstream video-generation and 3D reconstruction applications.
- 6. Conclusion: ViPE estimates camera poses, intrinsics, and depth maps from videos using dense and sparse bundle-adjustment constraints.The system leverages optical flow and keypoint tracking.
- 6. Conclusion: A depth alignment strategy provides consistent depth maps across frames.
- 6. Conclusion: ViPE shows superior performance across benchmarks covering static and dynamic, indoor and outdoor scenes.
- 6. Conclusion: ViPE has supported training-data annotation, conditional buffers for world generation, and feed-forward 3D Gaussian splatting reconstruction.The cited applications include Gen3C, Cosmos, and BTimer.