Source-linked AI summary
KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry
Renbiao Jin, Danping Zou, Wenxian Yu
TL;DR
Feature-based VIO depends on sparse tracking, yet classical KLT can fail under rapid motion and low texture. KLTNet replaces it with a lightweight coarse-to-fine tracker and learned weighting, improving odometry accuracy while maintaining real-time embedded performance.
Problem
Classical KLT relies mainly on local image patches and can become unreliable under rapid motion or low-texture conditions, affecting feature-based VIO state estimation.
Method
KLTNet combines low-resolution dense-flow initialization, triplet-patch refinement with a fixed reference patch, and differentiable-triangulation supervision of anisotropic observation weights.
Results
KLTNet improves tracking and odometry across VINS-Mono and OpenVINS evaluations, with VINS-Mono average absolute trajectory error reductions of 34% on EuRoC and 49% on TUM-VI while maintaining real-time performance.
Takeaways & Limitations
The tracker is a plug-and-play replacement for classical KLT in compatible KLT-based VIO systems and supports adaptive observation weighting.
Takeaways & Limitations
The coarse-to-fine tracker relies on coarse flow to initialize features within the local refinement capture range and assumes predominantly static scene points.
Abstract
from arXiv · showhide
Many feature-based visual-inertial odometry (VIO) systems rely on sparse feature tracking, whose accuracy and robustness directly affect state estimation. Classical KLT trackers rely primarily on local image patches and can become unreliable under rapid motion or in low-texture environments. We propose KLTNet, a lightweight learning-based, plug-and-play sparse feature tracker designed to replace classical KLT trackers in KLT-based VIO front ends. KLTNet follows a coarse-to-fine, dense-to-sparse architecture that combines low-resolution dense optical flow for robust global motion initialization with triplet-patch refinement for accurate and temporally consistent tracking. A fixed reference patch provides a stable anchor throughout each feature track and helps reduce accumulated tracking drift. In addition, KLTNet predicts anisotropic confidence weights supervised through differentiable multi-view triangulation, which can be used as observation weights in compatible VIO estimators. Experiments with VINS-Mono and OpenVINS on public benchmarks and a self-collected low-texture dataset demonstrate improved tracking and odometry accuracy over classical KLT, while maintaining real-time performance on an embedded platform.
I. INTRODUCTION
KLTNet is a lightweight, plug-and-play sparse tracker for KLT-based VIO, addressing KLT’s unreliability under rapid motion and low texture. It combines global-flow initialization, triplet-patch refinement, fixed reference patches, and learned observation weighting, improving odometry while retaining real-time performance.
- KLT becomes unreliable under rapid motion and low-texture conditions because it relies mainly on local image patches.
- KLTNet preserves a standalone sparse-tracker interface for existing KLT-based VIO systems while targeting robustness, accuracy, and temporal consistency.
- Its coarse-to-fine, dense-to-sparse design combines global dense-flow initialization with triplet-patch refinement and a fixed reference patch.
- Differentiable multi-view triangulation supervises anisotropic confidence weights that can support adaptive observation weighting in compatible VIO estimators.
- 34% lower average absolute trajectory error on EuRoC and 49% lower average absolute trajectory error on TUM-VI were achieved with VINS-Mono while maintaining real-time performance on a resource-constrained platform.
II. RELATED WORK
Related work spans lightweight sparse trackers, dense optical flow, multi-frame learned estimators, and learned observation weighting. KLTNet combines global dense-flow initialization with sparse triplet-patch refinement while remaining a tracker-level component.
- Lightweight VIO systems commonly use KLT to propagate sparse feature points because it is efficient and accurate for frame-to-frame tracking.
- Prior sparse-tracking methods improve robustness through pyramid processing or learned feature representations, whereas KLTNet additionally uses dense optical flow for global motion cues.
- RAFT and SEA-RAFT provide strong global motion cues, but KLTNet uses dense flow only for initialization and obtains final positions through triplet-patch refinement with a fixed reference patch.
- Multi-frame systems improve temporal consistency through joint front-end and back-end estimation but are closely coupled to complete odometry or SLAM pipelines.
- KLTNet predicts anisotropic confidence weights supervised through differentiable multi-view triangulation for sparse feature observations.
A. Coarse Dense Flow Initialization
KLTNet initializes sparse tracks with low-resolution dense flow, then refines them using triplet patches and motion priors. A fixed reference patch anchors each track to limit accumulated drift from recursive pairwise propagation.
- A. Coarse Dense Flow Initialization: Dense optical flow supplies global motion cues, while reduced-resolution initialization lowers computational cost in the coarse-to-fine tracker.
- A. Coarse Dense Flow Initialization: The coarse flow predicts an initial current-frame feature position from the previous tracked position.
- B. Fine Sparse Triplet-Patch Refinement: Triplet-patch refinement uses reference, previous, and current patches with flow priors to regress a fine offset and confidence weight matrix.
- A. Coarse Dense Flow Initialization: CoarseFlowNet downscales consecutive images, extracts feature maps, constructs a two-level 4D cost volume, and iteratively refines flow before sparse sampling.
- B. Fine Sparse Triplet-Patch Refinement: A fixed reference patch provides a long-term anchor that helps constrain accumulated tracking errors beyond recursive frame-to-frame matching.
C. Learning Confidence Weights via Differentiable Triangulation
KLTNet learns anisotropic observation weights by coupling tracked observations through differentiable multi-view triangulation. A shared 3D point and confidence-weighted reprojection refinement jointly supervise feature positions and weights.
- Each feature track shares one 3D point, while KLTNet predicts a positive-definite 2×2 matrix for directional observation weighting.
- The tracker recursively processes N consecutive frames, converting refined feature positions into normalized image coordinates before triangulation.
- DLT initializes the 3D point by stacking multi-view ray constraints and solving the resulting homogeneous system with SVD.
- Two damped Gauss–Newton iterations then refine the point using a confidence-weighted reprojection objective.
- The predicted weights are anisotropic observation weights rather than metrically calibrated uncertainty and can downweight unreliable local appearance.
- Backpropagation through differentiable triangulation jointly supervises predicted positions and weights, encouraging geometry-consistent observation weighting.
D. Training and Implementation Details
KLTNet is trained in two stages, with coarse flow learned first and the patch refiner subsequently trained on recursively propagated 24-frame tracks. Training exposes refinement to accumulated errors while avoiding full-chain backpropagation.
- KLTNet is trained on TartanAir in two stages: CoarseFlowNet first, followed by a frozen-CoarseFlowNet TriPatchRefiner trained on 24-frame sequences.
- Features are detected with GFTT in the first frame, and ground-truth tracks are generated from depth and camera poses after excluding tracks with insufficient parallax.
- Sequential propagation exposes the refiner to accumulated tracking errors, while detaching prior predictions prevents backpropagation through the full temporal chain.
- A fixed 31 × 31 reference patch is extracted at initialization, alongside patches from the previous and coarse current positions.
- The overall training objective combines offset supervision with triangulation loss using λtri = 320.
IV. EXPERIMENTS
The experiments integrate KLTNet into VINS-Mono and OpenVINS by replacing their native KLT trackers, and evaluate tracking and odometry across public and low-texture datasets. Runtime is also assessed on an embedded Jetson platform.
- KLTNet replaces the native KLT trackers in VINS-Mono and OpenVINS, representing optimization-based and filter-based VIO systems.
- Evaluation uses multiple public datasets and a self-collected low-texture dataset to validate the method.
- Feature tracking is evaluated on Replica, while odometry accuracy is evaluated on EuRoC and TUM-VI.
- Runtime and resource consumption are additionally measured on an NVIDIA Jetson AGX Orin embedded platform.
A. Evaluation of feature tracking
On Replica, KLTNet provides the strongest low-threshold tracking precision and the slowest temporal error growth among the evaluated trackers. Its stability is associated with the fixed reference patch used throughout each track.
- Replica contains 8 indoor RGB-D scenes with 2000-frame sequences divided into 200-frame subsequences for tracking evaluation.
- The baselines are OpenCV KLT, RAFT-Sparse, and SEA-RAFT-Sparse, which sample sparse positions from dense optical flow.
- Tracking Precision measures the percentage of observations below a pixel-error threshold, while Temporal Tracking Error summarizes errors across five temporal intervals.
- KLTNet achieves consistently higher tracking accuracy across thresholds, with especially strong precision at 1–3 pixels.
- SEA-RAFT-Sparse is less precise than KLT at low thresholds but surpasses KLT as error tolerance increases, indicating greater robustness to large errors.
- KLTNet’s median tracking error remains low and stable over time, with slower error accumulation attributed to its fixed reference patch.
B. Evaluation of monocular VIO
KLTNet improves monocular VIO accuracy over classical KLT across EuRoC, TUM-VI, and two VIO frameworks. Its performance is also competitive with learned odometry baselines while retaining the plug-in evaluation setup.
- Public-dataset evaluation: KLTNet reduces VINS-Mono average ATE by 34% on EuRoC and 49% on TUM-VI relative to classical KLT.On EuRoC, average ATE decreases from 0.187 to 0.123 meters.
- Public-dataset evaluation: In OpenVINS, KLTNet reduces average ATE by 33% on EuRoC and approximately 20% on TUM-VI.These results show improvements in both optimization-based and filter-based VIO frameworks.
- Baseline comparison: RAFT-Sparse and SEA-RAFT-Sparse do not consistently improve odometry because recursively chained flow errors can accumulate, whereas KLTNet combines robust initialization with precise refinement.SEA-RAFT-Sparse reduces EuRoC average ATE by 11% versus KLT, but remains less precise at low error thresholds.
- Baseline comparison: KLTNet achieves accuracy comparable to DVI-SLAM in OpenVINS while outperforming it in computational efficiency.The comparison is made against a dense learned odometry system using dense flow, IMU factors, and bundle adjustment.
C. Ablation Study
Ablations identify patch refinement, differentiable triangulation supervision, anisotropic weighting, point count, and the fixed reference patch as important design factors. The full system remains reliable with fewer tracked points, although ATE increases gradually.
- Patch refinement: Adding TriPatchRefiner to CoarseFlowNet substantially reduces ATE, showing that patch refinement contributes strongly to tracking performance.CoarseFlowNet alone preserves stability but produces imprecise tracking and higher ATE.
- Anisotropic confidence weights: Differentiable multi-view triangulation supervision improves over reprojection-based confidence supervision, while its anisotropic variant achieves the lowest ATE.The anisotropic triangulation variant also improves over the fixed-weight KLTNet configuration.
- Number of tracked points: Fewer tracked points cause a gradual increase in ATE, but the system remains reliable and reasonably accurate with as few as 25 points.The result is attributed to KLTNet’s tracking precision and confidence weights.
- Reference patch: Removing the fixed reference patch increases tracking error at short and long intervals, with a larger effect over long intervals.The resulting tracking inconsistency also raises VIO ATE in both VINS-Mono and OpenVINS.
D. Robustness Evaluation
KLTNet improves robustness beyond indoor benchmarks, including low-texture, complex-motion, and illumination-change settings. In the self-collected dataset, it achieves the lowest ATE across all evaluated sequences.
- KITTI-360 and UMA-VI: KLTNet reduces mean KITTI-360 relative translation error by 27.8% over KLT and has the lowest translation error on all eight sequences.It also has the lowest relative rotation error on seven sequences.
- Evaluation setup: KLTNet is evaluated on KITTI-360 and the illumination-change subset of UMA-VI within the same VINS-Mono configuration used for the public benchmark comparisons.The evaluation compares trackers rather than replacing the complete VIO system.
- Low-texture evaluation: On the self-collected low-texture dataset, KLTNet-VINS-Mono consistently achieves the lowest ATE across all sequences.Its advantages become clearer as motion complexity increases; DM-VIO fails to track in seq2 while KLT-VINS-Mono accumulates substantial drift.
E. Resource Consumption
The resource evaluation measures complete VINS-Mono front-end throughput and GPU memory for KLTNet and learned sparse-flow baselines. KLTNet is tested on both a desktop GPU and an embedded Jetson platform, with TensorRT acceleration on Jetson.
- Throughput and memory evaluation: KLTNet throughput is measured for the complete VINS-Mono visual front end across maximum track counts of 25, 50, 100, and 150.The comparison includes RAFT-Sparse, SEA-RAFT-Sparse, and end-to-end learned odometry systems as system-level reference points.
- Hardware and acceleration: KLTNet is evaluated on an NVIDIA A6000 PC and a Jetson AGX Orin embedded platform.The PC uses LibTorch without TensorRT, while the Jetson accelerates CoarseFlowNet and TriPatchRefiner with TensorRT.
- Embedded deployment: KLTNet maintains real-time tracking on the Jetson AGX Orin while providing improved accuracy and robustness across datasets and VIO frameworks.The conclusion frames this as embedded-platform performance rather than a claim about every deployment setting.