PaperlayerView exact revision on arXiv

Source-linked AI summary

NSFlow: End-to-End Differentiable Neuro-Symbolic Optical Flow for Visual Odometry

Yicheng Lin, Zhipeng Fei, Yuxiu Xu, WenDong Chen, Cong Li, Bin Han

arXiv:2609.06074v2cs.CV

TL;DR

VO/VIO depends on reliable inter-frame correspondence, yet optical-flow approaches face trade-offs among robustness, geometric consistency, and efficiency. The paper combines CNN features with a differentiable LK solver, using implicit differentiation for end-to-end training. In VIO tests, the method is best on average, with its advantage more pronounced under illumination disturbance.

  • Problem

    Optical flow is widely used for VO/VIO correspondence, but optimization-based flow is vulnerable to illumination changes and regression-based flow can lack geometric consistency and efficiency.

    Source: §I · p. 1

  • Method

    A CNN extracts image features for an LK solver that estimates sparse flow by minimizing descriptor differences, with implicit differentiation enabling end-to-end training.

    Source: §I · p. 2, 1) Encoder Network · p. 4, 2) Computation of Derivatives · p. 5

  • Results

    In VIO tests, the method is best on average, with its advantage more pronounced under small and large illumination disturbances and the lowest error on all sequences in those conditions.

    Source: 1) Experiments on the Enhanced EuRoC… · p. 11

  • Takeaways & Limitations

    The method can be inserted into VO/VIO systems, and training with simulated data alone generalizes across multiple real-world scenarios.

    Source: §V · p. 12

Abstract

from arXiv · show

Sparse optical flow provides stable inter-frame correspondence, playing a key role in Visual Odometry (VO) and Visual-Inertial Odometry (VIO). Classical optimization-based methods, such as Lucas-Kanade (LK), perform well under small displacements but are sensitive to large motions and illumination changes. Modern regression-based learning methods, while more robust in complex scenes, are often computationally heavy and lack explicit geometric consistency, making them less suitable for efficient VO/VIO front-ends. To bridge this gap, we propose a hybrid neuro-symbolic framework that combines the strengths of both paradigms. Our method uses a Convolutional Neural Network (CNN) to extract robust feature representations, which is fed into a differentiable LK optimizer to estimate optical flow in an end-to-end trainable manner. Through implicit differentiation, gradients are propagated across the iterative solver, enabling joint optimization of feature extraction and flow estimation. The resulting system integrates seamlessly into existing VO/VIO pipelines and runs in real-time on embedded platforms. Experiments show that our method outperforms conventional optimization-based flow in challenging conditions such as dynamic lighting and low texture, while also achieving higher accuracy and lower latency than purely regression-based alternatives. When deployed in a VIO system, our method demonstrates significant performance improvement, achieving an average error reduction of 42\% on challenging datasets while enhancing tracking stability. The code is publicly available.

I. INTRODUCTION

The introduction frames robust inter-frame correspondence as a VO/VIO challenge and proposes a neuro-symbolic method combining learned features with differentiable LK. The related work positions this approach between optimization-based and regression-based flow, both of which have limitations for odometry.

I. INTRODUCTION

  • Changing lighting, motion blur, and low-texture regions degrade inter-frame matches, threatening trajectory stability and practical VO/VIO deployment.Optical flow is widely used for correspondence, but its robustness can also suffer under illumination variation, large displacements, and real-time constraints.Source: §I · p. 1
  • A CNN feature extractor and an implicitly differentiated LK solver form an end-to-end trainable framework for sparse optical flow.The solver minimizes feature differences at warped locations, and implicit differentiation propagates gradients through its converged solution.Source: §I · p. 2

II. RELATED WORK

  • The related-work review covers deep learning-enhanced odometry, learned optical flow for VO/VIO, and differentiable optimization as three strands relevant to the proposed system.It frames the approach as addressing the trade-off among robustness, accuracy, and efficiency.Source: §II · p. 2

A. Deep Learning Enhanced Visual Odometry

  • Learned optical flow has served as an intermediate representation for pose estimation and supports strong generalization, but regression-based dense flow can lack geometric consistency and efficiency.Systems including DROID-SLAM, DPVO, TartanVO, and MAC-VO use optical flow before pose optimization.Source: A. Deep Learning Enhanced Visual… · p. 2

B. Learned Optical Flow for Visual Odometry

  • Optimization-based flow offers geometric consistency for pose estimation, whereas regression-based flow learns complex motion patterns but does not by itself ensure geometric consistency for VO.The review motivates learning flow from data while retaining precise geometric constraints.Source: B. Learned Optical Flow for Visual… · pp. 2–3

C. Differentiable Optimization

The paper combines a learned image encoder with an optimization-based LK flow layer, using implicit differentiation to train across the solver. The encoder produces descriptor maps, and the LK objective matches descriptors rather than pixel intensities.

C. Differentiable Optimization

  • Implicit differentiation computes gradients directly at the optimum, and the paper presents a differentiable LK formulation for optical flow estimation.Unlike unrolled differentiation, it does not track the full optimization path.Source: C. Differentiable Optimization · p. 3

III. METHOD

  • The method jointly estimates sparse optical flow with a neural encoder for descriptor maps and a differentiable LK layer.The method section covers the system framework, forward and backward solver passes, training losses, and VIO implementation.Source: §III · p. 3, A. Overview of the System Architecture · p. 3

A. Overview of the System Architecture

  • The system pairs a CNN descriptor-map encoder with an LK layer that iteratively estimates optical flow from those maps.The pipeline detects sparse keypoints in the first frame, extracts descriptors from consecutive frames, and computes flow from the descriptor maps.Source: A. Overview of the System Architecture · p. 3, Figure 2 · p. 4

1) Encoder Network:

  • The lightweight encoder preserves image resolution while producing descriptor maps, without explicitly specifying an optimal descriptor representation.Its stated design uses local image information rather than high-level features.Source: 1) Encoder Network · p. 3
  • The LK objective minimizes the distance between descriptors in consecutive frames instead of pixel-intensity differences, addressing settings where brightness constancy often fails.The passage names dynamic illumination, weak textures, and noise as such settings.Source: 1) Encoder Network · p. 4

2) Descriptor-Based LK Method:

The forward pass formulates LK flow estimation as a nonlinear least-squares problem over descriptor differences. Its first-order derivative approximation assumes small inter-frame flow.

B. Forward Pass of the Differentiable LK Layer

  • The LK solver uses Levenberg-Marquardt to minimize Euclidean distance between image descriptors rather than pixel-intensity differences.The paper describes the descriptor-based objective as a nonlinear optimization problem.Source: B. Forward Pass of the Differentiable… · p. 4

1) First-Order Derivatives of the Descriptor-Based Error Function:

  • The descriptor difference between the two images is defined as the error function used for flow estimation.The passage introduces this error function before deriving its first-order approximation.Source: 1) First-Order Derivatives of the… · p. 4
  • The first-order derivative approximation assumes the optical-flow increment between consecutive frames is small, δu ≈ 0.The derivation applies a Taylor expansion at δu = 0; descriptor gradients are taken along the x and y directions.Source: 1) First-Order Derivatives of the… · p. 4

2) The Iterative Levenberg-Marquardt Optimization:

The forward solver uses iterative Levenberg-Marquardt optimization, whose stopping and damping decisions are non-differentiable. The paper motivates implicit differentiation as a way to differentiate the locally optimal flow without modifying the original algorithm.

2) The Iterative Levenberg-Marquardt Optimization:

  • The LM solver iteratively updates flow using a Jacobian-based Hessian approximation and damping, but its convergence and damping decisions are non-differentiable.The damping factor increases when a step fails to reduce the objective and decreases when it succeeds.Source: 2) The Iterative Levenberg-Marquardt… · pp. 4–5
  • BA-Net addresses LM non-differentiability with fixed iterations and a learned damping update, but these changes can create training-inference inconsistency.The paper says its implicit differentiation method requires no modification to the original algorithm.Source: 2) The Iterative Levenberg-Marquardt… · p. 5

1) Implicit Function Theorem (IFT):

  • The Implicit Function Theorem gives derivatives of an implicitly defined function by differentiating its constraint, even without a closed-form function.The theorem requires the relevant partial derivative matrix to be invertible.Source: 1) Implicit Function Theorem (IFT) · p. 5

2) Computation of Derivatives:

The backward pass treats the locally optimal flow as implicitly defined by the descriptor maps. It uses the optical-flow estimation error to compute gradients for learning the descriptors.

2) Computation of Derivatives:

  • Optical-flow estimation error supplies the training signal for computing descriptor gradients through backpropagation.The method is intended to learn useful descriptors without a manually defined loss.Source: 2) Computation of Derivatives · p. 5
  • The objective is expressed in terms of descriptor maps and flow at keypoints, with each keypoint coordinate taken from image 1.The locally optimal flow estimate is obtained using the LM algorithm described earlier.Source: 2) Computation of Derivatives · p. 5
  • The locally optimal flow satisfies an implicit-function condition, allowing the function to be differentiated implicitly with respect to the descriptor maps.The function is expanded into components before applying the Implicit Function Theorem.Source: 2) Computation of Derivatives · pp. 5–6
  • The descriptor gradient with respect to flow is represented by the spatial gradient of the second image's descriptor map at the displaced keypoint.The derivatives are considered along the x and y optical-flow directions.Source: 2) Computation of Derivatives · pp. 5–6

D. End-to-end learning with Differentiable LK Layer

The section describes end-to-end training through a differentiable LK layer and integration of learnable optical flow into VINS-Fusion. The integrated system reports improved tracking accuracy, reduced drift, and greater robustness to illumination and motion variations.

D. End-to-end learning with Differentiable LK Layer

  • Training perturbs consecutive frames, initializes noisy keypoint flows, and refines correspondences in feature space with an LM solver and matching loss.The training correspondences come from ground-truth flow or a known geometric warp.Source: D. End-to-end learning with… · p. 6
  • Implicit differentiation propagates gradients through the LM optimization, jointly training the descriptor network and flow solver for the final objective.The differentiable LK layer supports direct end-to-end optimization rather than indirect descriptor supervision with manually designed losses.Source: D. End-to-end learning with… · p. 6

E. VIO System with Neuro-Symbolic Optical Flow

  • VINS-Fusion replaces hand-crafted LK tracking with descriptor-guided learnable LM flow, reporting improved tracking accuracy, reduced drift, and robustness to illumination and motion variations.The resulting flow associations are fused with IMU pre-integration in the back-end.Source: Figure 3 · p. 6, E. VIO System with Neuro-Symbolic… · p. 6

IV. EXPERIMENTS

The experiments examine optical-flow properties, compare sparse-flow methods, and evaluate integration into VINS-Fusion. They report improvements in pose accuracy and robustness from the VIO integration.

IV. EXPERIMENTS

  • The experimental section evaluates optical-flow properties, compares the proposed sparse flow with state-of-the-art methods, and tests its integration into VINS-Fusion.The planned analyses include spatial sparsity, accuracy under varying motion and illumination, precision, and computational efficiency.Source: §IV · p. 7
  • The study introduces shared training and implementation settings, correspondence and odometry benchmarks, and computing devices before presenting results.The settings are kept consistent across subsequent experiments.Source: §IV · p. 7
  • The VINS-Fusion evaluation reports consistent improvements in pose-estimation accuracy and robustness.Source: §IV · p. 7

A. Experimental Setup

Experiments use TartanAir for training and six public benchmarks for correspondence and odometry evaluation. The benchmark collection spans varied domains, platforms, and visual conditions.

A. Experimental Setup

  • The experiments use shared settings covering training data, network parameters, relative-pose datasets, odometry datasets, and computing devices.These settings remain consistent across the subsequent experiments.Source: A. Experimental Setup · p. 7

1) Training data:

  • TartanAir supplies photo-realistic indoor and outdoor training scenes with varied lighting, weather, and motion, alongside ground-truth depth, flow, and poses.Source: 1) Training data · p. 7

2) Training details:

  • Training uses 640 × 480 color pairs, up to 200 keypoints, and a differentiable LK layer unrolled for up to 50 iterations in an 11×11 window.The model is trained with Adam at 1 × 10−3 for 1000 epochs; the lowest-training-loss checkpoint is retained.Source: 2) Training details · p. 7

3) Evaluation Datasets:

  • Six public benchmarks cover synthetic and real domains, aerial and ground vehicles, mixed indoor/outdoor scenes, low texture, and underwater settings.The collection includes HPatches, TartanAir, EuRoC MAV, KITTI Odometry, UMA-VI, and AQUALOC.Source: 3) Evaluation Datasets · p. 7

B. Analysis of Optical Flow Properties

The analysis evaluates trackable regions, iterative convergence, and inference efficiency. It attributes observed robustness and convergence behavior in part to learned features that strengthen weak-texture gradients.

1) Trackability Analysis:

  • Trackability results show broader, more consistent recoverable regions, especially in low-texture or partially illuminated areas, than classical LK and descriptor-based methods.Table I reports median proportions at <1, <2, and <3 pixel error thresholds; no cell values are supplied here.Source: Table I · p. 8, 1) Trackability Analysis · p. 8
  • Learned descriptors extract edge cues, enhance weak-texture regions, and propagate gradients to surrounding areas, which likely improves convergence under large viewpoint changes.The visualization includes indoor and underwater weak-texture scenes.Source: 1) Trackability Analysis · p. 8, Figure 5 · p. 8

2) Convergence Behavior Analysis:

  • The proposed method reduces flow error faster and reaches lower steady-state error than LK and LET-NET from a 5-pixel initial error.The authors report that weak-texture enhancement and gradient propagation accompany the improved iterative optimization.Source: Figure 6 · p. 8, 2) Convergence Behavior Analysis · p. 8

3) Inference Efficiency Analysis:

  • The model uses 1.01 × 10−3M parameters and 0.27 GFLOPs, with 0.84 ms GPU and 5.21 ms CPU inference for 320×240 images.The paper reports the best inference efficiency among compared networks, but says direct efficiency comparison with classical end-to-end flow is not meaningful because its compute depends on matching difficulty.Source: 3) Inference Efficiency Analysis · p. 9, Table II · p. 9

C. Comparison with State-of-the-Art Methods

The proposed method outperforms other methods on viewpoint- and illumination-varying image pairs, then performs better on fundamental-matrix estimation across most consecutive-frame sequences. The comparisons assess both matching error and geometric accuracy.

1) Tracking Error:

  • The tracking-error experiment extracts 1,000 keypoints and compares matching against ground-truth homographies across image pairs.Learned descriptors use brute-force matching, while Harris keypoints use LK optical flow.Source: 1) Tracking Error · p. 9, C. Comparison with State-of-the-Art… · p. 9
  • The proposed method slightly outperforms LET-NET and clearly outperforms other methods on image pairs with viewpoint or illumination changes.HPatches’ texture-rich, clear images make differences between the two methods difficult to show clearly.Source: Figure 7 · p. 9, 1) Tracking Error · p. 9

2) Fundamental Matrix Error:

  • The fundamental-matrix comparison uses keypoint inlier rate and average feature-to-epipolar-line distance as error measures.The evaluation uses TartanAir test sequences and compares optical-flow, sparse-matching, and other approaches.Source: 2) Fundamental Matrix Error · p. 9
  • The proposed method performs better on fundamental-matrix estimation across most consecutive-frame sequences and significantly outperforms non-end-to-end methods with similar ideas.Learning-based keypoint methods and LightGlue do not significantly improve fundamental-matrix results and are better suited to large-baseline matching for 3D reconstruction.Source: Table III · p. 10, 2) Fundamental Matrix Error · p. 10

3) Visualization Experiment:

The experiments examine accumulated sparse-flow tracking error and VIO under added illumination changes. On enhanced EuRoC, the proposed optical-flow module gives the best average result without disturbance and the lowest errors across sequences under disturbances.

3) Visualization Experiment:

  • The visualization experiment contrasts sparse tracking with dense regression-based flow, whose per-pixel accuracy may be insufficient for VO tasks.The paper frames pixel-level tracking accuracy as more important for VO than a global image-wide flow error.Source: 3) Visualization Experiment · p. 10
  • Across 60 consecutive EuRoC frames, the proposed method maintains keypoint accuracy better than SEA RAFT during repeated tracking.The experiment tracks 20 initial keypoints, making cumulative flow errors visible as a proxy for the gradual accumulation of pose-estimation errors.Source: Figure 8 · p. 10, 3) Visualization Experiment · pp. 10–11

1) Experiments on the Enhanced EuRoC Dataset:

  • The enhanced EuRoC sequences introduce small or large illumination changes alongside the original sequences to test VIO under dynamic lighting.The perturbations apply absolute and relative pixel transformations.Source: 1) Experiments on the Enhanced EuRoC… · p. 11, Figure 9 · p. 11
  • On enhanced EuRoC, the proposed method is best on average without illumination disturbance and has the lowest error on every sequence under small or large disturbances.Under large disturbances, original VINS loses tracked points on multiple sequences and fails to estimate a complete trajectory; severe motion blur on V2 03 favors unmodified VINS.Source: 1) Experiments on the Enhanced EuRoC… · p. 11

2) Experiments on the UMA-VI Dataset:

On UMA-VI, the proposed method achieves the lowest average error across most sequences; on AQUALOC, it gives the best result on most sequences. The conclusion identifies local-minimum convergence and unmodeled flow uncertainty as limitations.

2) Experiments on the UMA-VI Dataset:

  • On UMA-VI, the proposed method achieves the lowest average error across the vast majority of sequences.The comparison includes classical, end-to-end learned, and hybrid VIO approaches; all use stereo mode with loop detection disabled.Source: 2) Experiments on the UMA-VI Dataset · p. 11, Table V · p. 12

3) Experiments on the AQUALOC Dataset:

V. CONCLUSION

  • The conclusion attributes the method’s end-to-end training to implicit differentiation through the optical-flow solver.It reports improved robustness while maintaining high accuracy in simple scenes, and notes that the method can be inserted directly into VO/VIO systems.Source: §V · p. 12
  • Performance is constrained by convergence to local minima, especially on challenging optical-flow datasets; the method also leaves flow uncertainty unmodeled.The conclusion identifies combination with regression-based methods, pose-constraint training, and uncertainty estimation as future directions.Source: §V · p. 12

Related papers

Loading 2609.06074v2…