PaperlayerView exact revision on arXiv

Source-linked AI summary

Discriminately Treating Motion Components Evolves Joint Depth and Ego-Motion Learning

Mengtan Zhang, Zizhan Guo, Hongbo Zhao, Yi Feng, Zuyi Xiong, Yue Wang, Shaoyi Du, Hanli Wang, Rui Fan

arXiv:2511.01502v1cs.CVcs.RO

TL;DR

Depth and ego-motion jointly determine supervision, yet ego-motion is often treated as auxiliary and motion types have distinct geometric properties. DiMoDE aligns optical axes and imaging planes to impose component-specific geometric constraints; it reports large ATE reductions on long-term odometry sequences but does not surpass KITTI depth-estimation SoTA.

  • Problem

    Ego-motion is often treated as auxiliary, despite jointly determining supervision with depth, while mixing motion types can overlook their distinct geometric properties and depth-independent rotational flow.

    Source: §I · p. 2, §IV · p. 4

  • Method

    DiMoDE aligns optical axes and imaging planes, then uses network outputs to impose geometric constraints on individual motion components and refine PoseNet outputs.

    Source: §I · p. 3

  • Results

    On four long-term odometry sequences, DiMoDE reduced ATE against the previous SoTA by 75.05%, 65.30%, 52.11%, and 54.56%, but did not surpass depth-estimation SoTA on KITTI.

    Source: D. Monocular Visual Odometry Results · p. 11, E. Monocular Depth Estimation Results · p. 12

  • Takeaways & Limitations

    The authors report that the geometric constraints enhance depth supervision and the robustness and stability of depth learning.

    Source: §I · p. 3, §VIII · p. 16

    DiMoDE still struggles with reliable predictions under extremely low illumination, and future work targets better generalizability in complex real-world environments.

    Source: §VII · p. 16, §VIII · p. 16

Abstract

from arXiv · show

Unsupervised learning of depth and ego-motion, two fundamental 3D perception tasks, has made significant strides in recent years. However, most methods treat ego-motion as an auxiliary task, either mixing all motion types or excluding depth-independent rotational motions in supervision. Such designs limit the incorporation of strong geometric constraints, reducing reliability and robustness under diverse conditions. This study introduces a discriminative treatment of motion components, leveraging the geometric regularities of their respective rigid flows to benefit both depth and ego-motion estimation. Given consecutive video frames, network outputs first align the optical axes and imaging planes of the source and target cameras. Optical flows between frames are transformed through these alignments, and deviations are quantified to impose geometric constraints individually on each ego-motion component, enabling more targeted refinement. These alignments further reformulate the joint learning process into coaxial and coplanar forms, where depth and each translation component can be mutually derived through closed-form geometric relationships, introducing complementary constraints that improve depth robustness. DiMoDE, a general depth and ego-motion joint learning framework incorporating these designs, achieves state-of-the-art performance on multiple public datasets and a newly collected diverse real-world dataset, particularly under challenging conditions. Our source code will be publicly available at mias.group/DiMoDE upon publication.

I. INTRODUCTION

Unsupervised joint learning of depth and ego-motion can avoid manual annotations, but mixing distinct motion flows can weaken supervision. The paper analyzes these flows and introduces geometric alignments and constraints to treat motion components separately.

I. INTRODUCTION

  • Joint monocular learning uses DepthNet to estimate dense depth and PoseNet to regress ego-motion, then trains both by minimizing photometric differences after rigid-flow warping.This paradigm avoids extensive manual annotations and can remain robust when 2D correspondences are sparse or unreliable.Source: §I · p. 1
  • Tangential translation provides inverse-depth positional shifts, radial translation adds perspective scaling, and rotation produces irregular depth-independent flow.Mixing tangential and radial cues can create inconsistent image gradients that may impede convergence and degrade depth estimation.Source: §I · pp. 2–3, Figure 1 · p. 2
  • Optical-axis and imaging-plane alignments reformulate joint learning into coaxial and coplanar forms, enabling separate geometric constraints for motion components and depth.The framework uses transformed correspondences and expected flow regularities to supervise translation components and constrain rotation.Source: §I · pp. 2–3

A. Monocular Visual Odometry

  • Earlier visual-odometry methods use hand-crafted correspondences or direct warping, while learning-based PoseNets and hybrid back-end optimization offer different robustness and resource trade-offs.Hybrid methods can outperform fully learning-based approaches but have storage and computational costs that hinder resource-constrained use.Source: A. Monocular Visual Odometry · p. 3
  • A simple ResNet-based PoseNet with DiMoDE achieves performance comparable to state-of-the-art geometric and hybrid approaches with lower storage and computational overhead.DiMoDE is presented as a general framework for joint depth and ego-motion learning across challenging conditions.Source: A. Monocular Visual Odometry · p. 3, §I · p. 3

B. Monocular Depth Estimation

Prior unsupervised monocular-depth methods rely on photometric consistency, which can become unreliable in adverse conditions. DiMoDE instead discriminates among motion components to create a robust constraint cycle, building on the standard joint depth-and-pose training setup.

B. Monocular Depth Estimation

  • Monocular-depth research has moved from supervised depth-ground-truth training toward unsupervised learning from monocular video to reduce dependence on extensive annotations.Other lines of work use discretized depth bins or vision foundation models for supervised estimation.Source: B. Monocular Depth Estimation · p. 3
  • DiMoDE targets adverse-condition robustness by discriminating motion components during supervisory-signal generation, rather than relying solely on per-pixel supervisory consistency.Earlier approaches add cues for occlusions and dynamic objects, while other methods adapt from daytime scenes to challenging environments using fixed data splits.Source: B. Monocular Depth Estimation · pp. 3–4

III. PRELIMINARIES

  • DepthNet predicts depth maps from target and source RGB images, while PoseNet estimates their relative ego-motion for rigid-flow image warping.The networks are trained jointly by minimizing photometric discrepancy between the target image and the warped source image.Source: §III · p. 4
  • The photometric reprojection loss combines appearance discrepancy with structural similarity, using an empirically set SSIM weight of 0.85.The loss serves as the supervisory signal for both DepthNet and PoseNet.Source: §III · p. 4

IV. DISCRIMINATIVE TREATMENT OF MOTION COMPONENTS

The section argues that treating rotation, tangential translation, and radial translation separately preserves useful flow regularities and avoids cross-component interference. It uses transformed optical flows and geometric constraints to refine motion components and support depth learning.

IV. DISCRIMINATIVE TREATMENT OF MOTION COMPONENTS

  • Existing approaches either exclude rotation from depth supervision or mix motion types, overlooking distinct translational-flow regularities and leaving decoupled rotation errors unoptimized.The section notes that this can add computational and storage overhead and allow rotational errors to degrade overall performance.Source: §IV · p. 4
  • The method isolates rotation and processes tangential and radial translations separately, using PoseNet outputs to align camera orientations, optical axes, and imaging planes.These alignments remove irregular rotational flows and generate regular translational flows.Source: §IV · pp. 4–5, Figure 2 · p. 5
  • Pure rotation produces irregular rigid flow determined by rotation and camera intrinsics, whereas tangential and radial translations produce more regular flows.Tangential flow varies with depth; radial flow points toward or away from the principal point and depends on depth and pixel coordinates.Source: §IV · p. 5
  • Mixing tangential and radial flows obscures their regularities, while gradients for one translation component are influenced by the other and can impair PoseNet convergence.Depth receives smaller gradients near the principal point, hindering consistent image-wide optimization.Source: §IV · pp. 5–6
  • Decomposed flows support geometric constraints that independently refine depth and each translational component after rotational flows are removed.The two constraints also guide rotation estimation while supervising tangential and radial translations independently.Source: §IV · p. 6
  • PoseNet prediction errors introduce rotational and cross-translation deviations into the decomposed flows, weakening the constraints based on their expected regularities.The section proposes jointly minimizing deviations between decomposed flows and expected flows using two geometric constraints.Source: §IV · p. 6
  • The new dataset includes pronounced rotational motion and frequent camera shake to test robustness beyond driving datasets with limited rotation-estimation challenges.The passage notes that radial translation typically dominates tangential translation, so even minor rotational deviation can produce noticeable tangential-translation deviation.Source: §IV · p. 6

V. DIMODE

The method recasts motion-component removal as camera alignment and incorporates those alignments into joint learning to constrain both networks.

V. DIMODE

  • Removing rotation and tangential translation is equivalent to aligning the source and target cameras’ optical axes.Removing rotation and radial translation is equivalent to aligning their imaging planes.Source: §V · p. 6
  • DiMoDE incorporates the optical-axis and imaging-plane alignment processes into the joint learning framework.The alignments introduce explicit geometric constraints.Source: §V · p. 6
  • The explicit geometric constraints refine outputs from both DepthNet and PoseNet.The passage presents this refinement as the purpose of incorporating the alignment processes into joint learning.Source: §V · p. 6

A. Framework Overview

DiMoDE decomposes PoseNet motion estimates to align cameras and transform optical flows. It then uses geometric constraints to refine pose components and, once alignment is sufficiently accurate, depth.

A. Framework Overview

  • PoseNet motion is decomposed into rotation, tangential translation, and radial translation to align camera orientation, optical axes, and imaging planes.DepthNet supplies depth estimates, while FlowNet generates dense correspondences.Source: A. Framework Overview · p. 6, Figure 4 · p. 7
  • Optical flows are transformed through the alignments, and deviations from expected pure radial and tangential flows constrain refinement of PoseNet components.The framework overview identifies two sets of geometric constraints from the transformed flows.Source: A. Framework Overview · p. 6, Figure 4 · p. 7
  • After optical axes and imaging planes are sufficiently aligned, depth and each translation component can be mutually determined through decomposed flows to supervise DepthNet.These constraint cycles provide robust depth supervision under adverse conditions.Source: A. Framework Overview · pp. 6–7, Figure 4 · p. 7

B. Optical Axis and Imaging Plane Alignments

The method transforms original optical-flow correspondences through optical-axis and imaging-plane alignments rather than repeatedly warping images. This produces flows for the aligned views.

B. Optical Axis and Imaging Plane Alignments

  • Repeated image warping can add computational overhead, holes, and artifacts, so the method transforms original optical flow through the alignments instead.The paper links those artifacts to less reliable correspondence generation and weaker geometric constraints.Source: B. Optical Axis and Imaging Plane… · p. 7
  • Target-to-source optical flow generates pixel correspondences once for the alignment procedure.The source-view depth map is warped into the target view to produce a pixel-aligned depth map for backprojection.Source: B. Optical Axis and Imaging Plane… · p. 7
  • The aligned flows are derived by backprojecting the correspondences, applying the alignment processes, and projecting back into pixel coordinates.The passage identifies these resulting flows as the optical flows for the aligned imaging plane and optical axis.Source: B. Optical Axis and Imaging Plane… · p. 7

C. Geometric Constraints from Motion Components

The method uses motion-specific rigid-flow geometry to align camera axes and imaging planes, then imposes complementary constraints linking depth and translation. These constraints are intended to strengthen depth learning in challenging conditions.

C. Geometric Constraints from Motion Components

  • Tangential flows should have a uniform direction, while radial flows should point toward or away from the principal point; their constraints use direction consistency.The tangential-flow vectors maintain a consistent angle with a global reference vector, while radial-flow vectors are parallel to the direction from each pixel to the principal point.Source: C. Geometric Constraints from Motion… · p. 7
  • Radial translation is optimized for imaging-plane alignment, tangential translation for optical-axis alignment, and rotation is constrained by both.The constraints are computed over valid pixels in static, non-occluded regions.Source: C. Geometric Constraints from Motion… · p. 8
  • The alignments reformulate joint learning into coaxial and coplanar forms, where depth and ego-motion satisfy simplified geometric relations.The forms are analogous to axial-motion image pairs and stereo image pairs along horizontal and vertical directions.Source: C. Geometric Constraints from Motion… · p. 8
  • At each pixel, three depth-to-translation ratios yield geometric depth values that are checked against DepthNet and can recover translation in closed form.The recovered translation components are expected to match PoseNet estimates, and the inverse computation does not require iterative optimization.Source: C. Geometric Constraints from Motion… · p. 8
  • The two constraint cycles aggregate depth errors across valid pixels, complementing pixel-level constraints used by prior methods,,.The paper proposes that these cycles can help prevent substantial DepthNet degradation when per-pixel supervision is unreliable, including at night or in adverse weather.Source: C. Geometric Constraints from Motion… · p. 8
  • The overall loss combines alignment and motion-component losses with local-structure refinement and photometric reprojection losses.The loss is weighted as λ1(Laxi + Lpla) + λ2(Lrad + Ltan) + λ3Lloc + Lpho.Source: C. Geometric Constraints from Motion… · p. 8

VI. EXPERIMENTS

The experiments compare the method with prior approaches on visual odometry and depth estimation, then use ablations to assess its designs and hyperparameter selection.

VI. EXPERIMENTS

  • The evaluation compares the method with prior state-of-the-art approaches in monocular visual odometry and depth estimation.The comparison uses six public datasets and a newly created real-world dataset, as described in the experiments overview.Source: §VI · p. 8
  • Ablation studies assess the efficacy of designs based on discriminative motion-component treatment.The experiments overview also identifies hyperparameter selection as part of these studies.Source: §VI · p. 8
  • The experiments include a newly created real-world dataset intended to evaluate performance in complex scenarios.The overview also lists six public datasets for performance comparison.Source: §VI · p. 8

A. Implementation Details

The implementation combines established depth, pose, and flow networks with progressive training. The evaluation spans public datasets and a new handheld real-world dataset for odometry.

A. Implementation Details

  • DiMoDE is evaluated with Lite-Mono and D-HRNet depth networks, ResNet-18-based or spatial-clue PoseNets, and RAFT for dense correspondences.Lite-Mono and the basic ResNet-based PoseNet are used for most experiments because D-HRNet is more computationally costly and the alternative PoseNet does not support real-time inference.Source: A. Implementation Details · pp. 8–9
  • Training uses three consecutive frames, AdamW, ImageNet-pretrained depth and pose encoders, and a FlowNet pretrained on FlyingChairs.The reported setup uses an initial learning rate of 10^-4 and weight decay of 10^-2.Source: A. Implementation Details · p. 9
  • Training progresses from photometric and local losses, to alignment constraints, then to tangential and radial constraints.The three stages last 5, 10, and 15 epochs respectively; FlowNet is frozen in Stage 2.Source: A. Implementation Details · p. 9

B. Datasets

  • KITTI Odometry training uses sequences 00–08, with 09–10 as the standard test set and 11–21 included for generalization and convergence evaluation.Pseudo-ground-truth trajectories from ORB-SLAM3 (Stereo) are used for sequences 11–21, which lack public ground-truth trajectories.Source: B. Datasets · p. 9
  • MIAS-Odom contains six handheld sequences, three indoor and three outdoor, totaling 11,608 images captured under challenging conditions.The synchronized sensor system includes binocular cameras, a Livox MID-360 LiDAR, and an IMU; trajectories are obtained with LiDAR-inertial odometry.Source: B. Datasets · p. 9, Figure 5 · p. 9
  • Depth evaluation uses KITTI, DDAD, and nuScenes, while Make3D and DIML are used only to assess generalizability.The latter datasets lack stereo pairs and monocular sequences for unsupervised training.Source: B. Datasets · p. 9

C. Evaluation Metrics

The section defines pose and depth metrics, then evaluates DiMoDE on KITTI test, long-term, and finetuning sequences. It reports strong learning-based odometry results, with real-world generalization dependent on the challenging sequences being included in training.

C. Evaluation Metrics

  • Pose performance is measured by average translational error, average rotational error, and ATE; depth uses Abs Rel, Sq Rel, RMSE, RMSE log, and threshold accuracies.The KITTI odometry comparison table distinguishes stereo and monocular training and marks the best results within each method category.Source: C. Evaluation Metrics · p. 9, Table I · p. 10

D. Monocular Visual Odometry Results

E. Monocular Depth Estimation Results

The depth evaluation finds DiMoDE competitive but not best on KITTI, while it outperforms leading methods on DDAD and nuScenes. Zero-shot evaluations and cross-dataset tests also examine how performance varies across more diverse conditions and training data.

E. Monocular Depth Estimation Results

  • On KITTI, DiMoDE achieves competitive depth-estimation performance but does not surpass existing SoTA approaches.The authors note that clear daytime scenes and few close-range objects near the principal point limit the demonstration of its intended advantages.Source: E. Monocular Depth Estimation Results · p. 12, Table V · p. 13
  • On DDAD and nuScenes, DiMoDE significantly outperforms previous leading methods, including robust depth methods on nuScenes.The evaluation describes these datasets as containing varying illumination, adverse weather, and structurally complex environments; the comparison tables cover DDAD and nuScenes.Source: E. Monocular Depth Estimation Results · pp. 12, 14, Table VI · p. 13, Table VII · p. 13
  • The authors link the stronger challenging-dataset results to global geometric constraints that complement pixel-level supervision in DepthNet training.Their discussion contrasts this constraint cycle with methods relying exclusively on pixel-level supervisory signals.Source: E. Monocular Depth Estimation Results · pp. 13–14
  • For zero-shot DDAD evaluation, models trained on nuScenes generalize better than KITTI-trained models and can perform comparably to models trained directly on DDAD.The authors attribute this pattern to nuScenes’ greater environmental complexity and diversity.Source: E. Monocular Depth Estimation Results · p. 14, Table VIII · p. 14
  • In zero-shot evaluations on Make3D and DIML using KITTI-trained models, DiMoDE has slightly better quantitative metrics and avoids overly distant predictions near the principal point in reported examples.The authors also report finer predictions in texture-rich regions and greater reliability in texture-less areas than the baseline.Source: E. Monocular Depth Estimation Results · pp. 14–15, Table IX · p. 14, Figure 13 · p. 15

F. Ablation Studies and Hyperparameter Selection

The ablations report gains from the proposed geometric constraints for pose and depth refinement, smoother convergence, and relative robustness to moderate hyperparameter changes. The discussion identifies low-illumination and unseen-sequence generalization as limitations, while the conclusion reports robust results across seven datasets and identifies future work.

F. Ablation Studies and Hyperparameter Selection

  • Removing either pose constraint degrades translation and rotation accuracy; the two constraints play complementary roles in constraining rotation.The reported translational refinement depends on well-constrained rotation; an alternative PoseNet architecture also shows consistent performance gains with the framework.Source: F. Ablation Studies and Hyperparameter… · p. 15, Table X · p. 15
  • DiMoDE significantly outperforms two prior depth-constraint approaches across three datasets, and retains gains with an alternative DepthNet architecture.The comparison is with cross-task consistency and contextual-geometric depth consistency losses.Source: F. Ablation Studies and Hyperparameter… · p. 15, Table XI · p. 15
  • DiMoDE converges more smoothly than the baseline, and ablations indicate each training component contributes to suppressing validation oscillations and improving performance.The figure compares validation curves for the baseline, DiMoDE, and ablation variants.Source: Figure 14 · p. 16, F. Ablation Studies and Hyperparameter… · p. 16
  • The default weights λ1 = 0.05 and λ2 = 0.1 perform best across both tasks; moderate adjustments from 0.05 to 0.2 yield comparable results.Performance degrades at 0.01 and 1, attributed respectively to insufficient gradient strength and geometric-loss dominance that destabilizes training.Source: F. Ablation Studies and Hyperparameter… · p. 16, Figure 14 · p. 16

VII. DISCUSSION

  • DiMoDE remains unreliable under extremely low illumination and does not generalize effectively to unseen real-world sequences.The discussion attributes the latter limitation to greater environmental and motion complexity in the collected dataset than in conventional driving scenarios.Source: §VII · p. 16

VIII. CONCLUSION AND FUTURE WORKS

  • Experiments across six public datasets and a newly collected dataset report robust learning for depth and visual odometry; future work targets illumination robustness and complex-scene generalization.The authors also plan to explore unsupervised learning across diverse scenes for scalable model training.Source: §VIII · p. 16

Related papers

Loading 2511.01502v1…