PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Patch Visual Odometry

Zachary Teed, Lahav Lipson, Jia Deng

arXiv:2208.04726v2cs.CV

TL;DR

Deep VO methods improve robustness but dense flow is computationally expensive, limiting practical use. DPVO instead tracks sparse image patches with a recurrent update operator and differentiable bundle adjustment, achieving strong accuracy with substantially lower runtime and memory.

  • Problem

    Prior VO systems face frequent failures, while dense-flow methods such as DROID-SLAM impose substantial computational and memory costs.

  • Method

    DPVO uses deep feature-based sparse patches, a recurrent network for patch correspondence, and differentiable bundle adjustment.

  • Results

    DPVO outperforms prior work across common benchmarks while running 1.5-8.9x faster and using 57-29% of the memory.

  • Takeaways & Limitations

    Sparse patch-based correspondence can provide both efficiency and robustness without relying on dense flow.

  • Takeaways & Limitations

    The method assumes constant depth across each image patch, representing the patch as a frontoparallel plane.

Abstract

from arXiv · show

We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular Visual Odometry (VO). DPVO uses a novel recurrent network architecture designed for tracking image patches across time. Recent approaches to VO have significantly improved the state-of-the-art accuracy by using deep networks to predict dense flow between video frames. However, using dense flow incurs a large computational cost, making these previous methods impractical for many use cases. Despite this, it has been assumed that dense flow is important as it provides additional redundancy against incorrect matches. DPVO disproves this assumption, showing that it is possible to get the best accuracy and efficiency by exploiting the advantages of sparse patch-based matching over dense flow. DPVO introduces a novel recurrent update operator for patch based correspondence coupled with differentiable bundle adjustment. On Standard benchmarks, DPVO outperforms all prior work, including the learning-based state-of-the-art VO-system (DROID) using a third of the memory while running 3x faster on average. Code is available at https://github.com/princeton-vl/DPVO

1. Introduction

Monocular VO must estimate camera pose and build an environmental map from video, but prior systems often fail or incur substantial computation. DPVO uses sparse patch tracking and differentiable optimization to improve efficiency while preserving accuracy and robustness.

  • Monocular VO estimates the camera’s 6-DOF pose at every frame while simultaneously building an environmental map.
  • Prior direct and indirect VO systems frequently fail because of moving objects, lost feature tracks, and poor convergence.
  • DROID-SLAM improves VO accuracy with dense flow but averages 40FPS while using 8.7GB GPU memory on an RTX-3090.
  • DPVO achieves lower average error than prior work while running 1.5-8.9x faster and using 57-29% of the memory.
  • DPVO combines deep feature-based patches, recurrent sparse trajectory tracking, and differentiable bundle adjustment for end-to-end reliable matching.
  • Patch-based correspondence can improve both efficiency and robustness over dense flow by reallocating resources toward matching accuracy.

2. Related Work

VO estimates motion and scene structure from video, while SLAM adds global corrections such as loop closure. Related systems range from classical sparse optimization to deep dense-flow methods, with different accuracy and computational trade-offs.

  • VO estimates robot state from video, whereas SLAM additionally corrects accumulated drift through loop closure and related global techniques.
  • Monocular VO is commonly formulated as nonlinear least-squares optimization within a factor-graph MAP framework.
  • DPVO borrows DSO’s patch representation and reprojection objective but replaces intensity residuals with neural predictions and automatic outlier rejection.
  • Unlike DROID-SLAM’s dense motion fields, DPVO selects sparse patches and maintains relatively constant runtime per frame.
  • DPVO provides 1.5-8.9x faster inference than DROID-SLAM.
  • Prior differentiable bundle-adjustment approaches include BA-Net and neural refinements atop COLMAP, but they differ in patch correspondence or standalone 3D reconstruction.

3. Approach

DPVO represents scenes with sparse image patches and uses a recurrent update operator with differentiable bundle adjustment to jointly refine patch trajectories, depths, and camera poses. Its design combines feature-based patch matching, dynamic patch graphs, and supervised pose and flow refinement.

  • Preliminaries: DPVO represents a scene with camera poses and square image patches, using inverse depth and pixel coordinates for patch geometry.Each patch is modeled with constant depth, forming a frontoparallel plane in its source frame.
  • Patch Graph: A dynamic bipartite patch graph connects each patch to nearby frames, with connected reprojections forming the patch trajectory.As new frames arrive, frames and patches are added while old ones are removed.
  • Approach Overview: The approach samples patches, estimates their 2D motion across connected frames, and solves for depths and camera poses through a recurrent network and differentiable optimization layer.This retains the structure of classical visual odometry while replacing its core estimation steps with learned recurrent updates and differentiable bundle adjustment.
  • Feature and Patch Extraction: Residual networks extract matching and context features, while randomly sampled patches crop corresponding feature regions through bilinear interpolation.Patch features compute visual similarities on the fly rather than using precomputed correlation volumes.
  • Update Operator: The recurrent update operator aggregates correlation information across patch-graph edges, updates hidden states, predicts trajectory revisions, and applies bundle adjustment to refine poses and patch depths.Its components include correlation, temporal convolutions, softmax aggregation, a transition block, a factor head, and differentiable bundle adjustment.
  • Training and Supervision: Supervised training uses pose and induced optical-flow losses, with intermediate update outputs supervised during optimization.Training samples trajectories whose frame-to-frame optical flow magnitude lies between 16px and 72px, targeting difficult but not impossible instances.

4. Experiments

DPVO is evaluated across several monocular VO and SLAM benchmarks, where it combines strong accuracy with substantially improved runtime and memory efficiency. Ablations examine patch representations, update-operator components, and patch-centroid selection.

  • Benchmark Results: DPVO achieves the lowest average error among prior VO systems across the evaluated benchmarks and outperforms full SLAM systems on some benchmarks.The evaluation covers TartanAir, TUM-RGBD, EuRoC, and ICL-NUIM, using median results across repeated trials.
  • TartanAir: 0.80 AUC versus 0.71 for DROID-SLAM is reported on the TartanAir validation split.The comparison uses the [0, 1]m error window.
  • TartanAir: 40% lower error than DROID-SLAM and 64% lower error than DROID-VO are reported on the TartanAir test split.Results are from the ECCV 2020 SLAM competition test split.
  • EuRoC: 43% lower average error than DROID-VO is reported on EuRoC, while the 120FPS system outperforms DROID-VO on most videos.The 60-FPS system outperforms prior work on the majority of EuRoC sequences.
  • Efficiency: 1.5x faster average runtime and 4.3x faster worst-case runtime are reported for the 60-FPS variant, while the 120-FPS variant reaches 3x and 8.9x.The worst-case comparison uses the 5th-percentile runtime.
  • Efficiency: DPVO uses 4.9GB in its Default configuration and 2.5GB in its Fast configuration, compared with 8.7GB for DROID-VO.Memory measurements were obtained with nvidia-smi on EuRoC.
  • Ablations: Patch features outperform point features, removing 1D-Convolution or Softmax-Aggregation degrades accuracy, and random patch-centroid selection performs best.These findings come from ablations on the TartanAir validation split.

5. Conclusion

DPVO is presented as an accurate and efficient deep visual odometry system based on sparse patches. It runs at 60–120 FPS with minimal memory and outperforms prior classical and learned methods on several benchmarks.

  • Conclusion: DPVO uses a sparse patch representation and runs at 60–120 FPS with minimal memory requirements.The conclusion reports outperformance on EuRoC, TUM-RGBD, the TartanAir ECCV 2020 competition, and ICL-NUIM.

A. Results on ICL-NUIM [19]:

On ICL-NUIM, DPVO outperforms prior work on most sequences while maintaining lower average error than DROID-SLAM and SVO.

  • ICL-NUIM: 51% lower average error than DROID-SLAM and 32% lower average error than SVO are reported for the faster DPVO model.The benchmark contains indoor scenes with repetitive or monochrome textures, and the faster and default DPVO systems perform similarly.

B. Stable Runtime

DPVO maintains a relatively stable runtime because it uses a simple keyframing mechanism. Treating all incoming frames as keyframes simplifies processing and keeps frame rate approximately constant, but is sub-optimal during slow or still motion.

  • Stable Runtime: DPVO’s default configuration averages 60FPS and rarely drops below 50FPS, with runtime remaining relatively constant compared with prior systems.Its runtime stability is attributed to the comparatively simple keyframing mechanism.
  • Keyframing: DPVO treats all incoming frames as keyframes and removes redundant frames later using motion from the estimated pose.This design keeps frame rate approximately constant regardless of camera motion.
  • Keyframing: The keyframing design is sub-optimal in speed during very slow or still camera movement.The limitation is explicitly noted for settings such as EuRoC.

C. ATE Error Metric

DPVO evaluates trajectories with the standard average trajectory error metric after similarity alignment to account for scale and SE(3) gauge freedoms.

  • ATE, or average trajectory error, is the standard metric used to compare predicted and ground-truth VO trajectories.
  • Similarity transformation alignment accounts for scale and SE(3) gauge freedoms before computing the metric.
  • The metric is computed using the EVO library.

D. Additional Training Details

DPVO is trained on synthetic TartanAir data matching DROID-SLAM’s dataset parameters, while its visualization system provides low-overhead interactive reconstructions.

  • Additional Training Details: DPVO is trained entirely on TartanAir, a synthetic dataset also used by previous VO systems.TartanAir covers indoor and outdoor scenes with varied lighting and weather, and provides depth and camera pose annotations.
  • Additional Training Details: TartanAir depth and camera pose annotations enable optical-flow generation by re-projecting depth using camera poses.
  • Additional Training Details: All dataset parameters are identical to those used in DROID-SLAM.
  • Additional Training Details: The visualizer reads directly from PyTorch tensors and slows the full system by approximately 10%.It uses the Pangolin library and avoids unnecessary CPU-to-GPU memory copies.
  • Additional Training Details: Figure C presents an interactive reconstruction on the ETH-3D dataset.

F. Bundle Adjustment Layer

DPVO uses a sparse bundle-adjustment implementation and recurrent edge updates to refine flow correspondences, confidence weights, camera poses, and patch depths.

  • Bundle Adjustment Layer: Because DROID-SLAM uses dense flow, DPVO implements an optimized sparse CUDA version of bundle adjustment.
  • Bundle Adjustment Layer: Accuracy on EuRoC quickly saturates after 96 tracked patches, exposing a speed–performance trade-off as patch count increases.
  • Bundle Adjustment Layer: The update operator correlates reprojected patches with matching-feature pyramids using bilinearly sampled 7 × 7 neighborhoods.Each sampled feature has dimension D = 128, and dot products produce matching correlation features.
  • Bundle Adjustment Layer: Learnable layers predict 2D flow revisions, confidence weights, and hidden-state updates for each patch-graph edge.
  • Bundle Adjustment Layer: The predicted factors guide bundle adjustment, which updates camera poses and patch depths.

H. Network Architecture

DPVO’s architecture combines separate context and matching feature extractors with edge-wise recurrent processing that predicts flow updates and confidence factors for bundle adjustment.

  • Network Architecture: The update operator uses 1D convolutions, message passing through Softmax-Aggregation, and a Transition Block.
  • Network Architecture: For each patch-graph edge, the operator predicts a 2D flow update, a confidence weight in R2, and an updated hidden state.
  • Network Architecture: The predicted edge factors are passed to bundle adjustment to update camera poses and patch depths.
  • Network Architecture: The factor head predicts confidence weights bounded to (0, 1) alongside 2D flow updates.
  • Network Architecture: Context and matching features use the same extractor architecture, with output dimensions D = 384 and D = 128 respectively.Matching features use instance normalization, whereas context features use no normalization.
  • Network Architecture: The architecture overview excludes non-learnable correlation and bundle-adjustment layers.
Loading 2208.04726v2…