Source-linked AI summary
GCNv2: Efficient Correspondence Prediction for Real-Time SLAM
Jiexiong Tang, Ludvig Ericson, John Folkesson, Patric Jensfelt
TL;DR
GCNv2 addresses the need for efficient, accurate visual odometry on embedded platforms for vision-based SLAM. It modifies GCN for single-frame prediction and binary descriptors, retaining comparable accuracy while enabling real-time GCN-SLAM on a Jetson TX2 and drone positioning.
Problem
Visual odometry is a cornerstone of vision-based SLAM, but GCN was prohibitively expensive for real-time use on embedded hardware.
Method
GCNv2 improves GCN efficiency through single-frame prediction, integrated descriptor binarization, and an ORB-compatible binary descriptor format.
Results
GCNv2 retains comparable accuracy to GCN, reduces inference time, and enables GCN-SLAM to run in real time on embedded hardware such as a Jetson TX2.
Takeaways & Limitations
GCNv2 can replace ORB in feature-based SLAM systems and supports robust real-time positioning of a flying drone.
Takeaways & Limitations
GCNv2 is trained for projective geometry rather than generic feature matching, and outdoor performance was not investigated further.
Abstract
from arXiv · showhide
In this paper, we present a deep learning-based network, GCNv2, for generation of keypoints and descriptors. GCNv2 is built on our previous method, GCN, a network trained for 3D projective geometry. GCNv2 is designed with a binary descriptor vector as the ORB feature so that it can easily replace ORB in systems such as ORB-SLAM2. GCNv2 significantly improves the computational efficiency over GCN that was only able to run on desktop hardware. We show how a modified version of ORB-SLAM2 using GCNv2 features runs on a Jetson TX2, an embedded low-power platform. Experimental results show that GCNv2 retains comparable accuracy as GCN and that it is robust enough to use for control of a flying drone.
I. INTRODUCTION
The paper targets efficient visual odometry for indoor drones, where deep feature extraction and matching must support robust real-time SLAM on embedded hardware. GCNv2 improves GCN’s efficiency while retaining accuracy, using binary descriptors compatible with existing SLAM systems.
- Motivation: Visual odometry estimates relative motion from visual information and forms a cornerstone of vision-based SLAM for mobile robotics.The method uses an RGB-D sensor, making scale directly observable and avoiding visual-inertial fusion or learned depth inference.
- Problem: GCN’s multi-frame matching and computational demands make real-time operation on embedded drone hardware difficult.The paper identifies deep-learning integration in performance-constrained SLAM as an open problem.
- Approach: GCNv2 improves computational efficiency while maintaining GCN’s high precision by predicting from a single frame at a time.This design addresses the multi-frame setup issue identified in GCN.
- Results: GCNv2 retains comparable accuracy to GCN while reducing inference time and improving motion estimation over related deep-learning feature extractors.
- Integration: Binary descriptors greatly accelerate matching and share ORB’s descriptor format, allowing direct use in systems such as ORB-SLAM2 and SVO2.
- Deployment: GCN-SLAM runs in real time on embedded low-power hardware such as the Jetson TX2 and handles situations where ORB-SLAM2 fails.
II. RELATED WORK
The related work spans direct, indirect, and semi-direct visual odometry and SLAM, alongside learned correspondence methods. GCN differs by tailoring keypoints and descriptors specifically to camera-motion estimation and by motivating a higher-throughput variant.
- VO and SLAM: Direct visual odometry estimates motion by aligning image frames using pixel intensities.
- VO and SLAM: Indirect methods extract keypoints, match their descriptors across frames, and use geometric constraints to estimate motion.ORB-SLAM2 is identified as state of the art in this category, while ORB’s binary descriptor enables high-performance matching.
- VO and SLAM: Semi-direct methods occupy an intermediate position, combining sparse or semi-dense strategies for motion estimation.SVO2 is described as sparse and capable of running at hundreds of Hertz.
- Deep learning methods: Recent learned mapping systems use predicted depth, normals, or learned representations for reconstruction and scale-drift reduction.
- Deep correspondence matching: Learned correspondence methods commonly use metric learning, improved invariances, or synthetic self-supervised samples to train image features.
- GCN: GCN targets motion estimation directly and reports improved performance over more general learned keypoint extractors, motivating the higher-throughput GCNv2.
III. GEOMETRIC CORRESPONDENCE NETWORK
GCNv2 revises GCN into a more efficient single-view network for real-time SLAM on embedded hardware. It predicts keypoints and descriptors at lower resolution and achieves substantially higher deployment rates with a smaller variant.
- Design goals: The GCNv2 design targets real-time SLAM on embedded hardware by revising the network structure and training a binarized feature descriptor.
- Limitations of GCN: GCN’s large architecture requires powerful hardware and cannot run in real time on platforms such as the Jetson TX2.
- Network structure: GCNv2 predicts keypoint-confidence maps and dense descriptor maps from a single image at lower resolution.This simplified structure is inspired by SuperPoint and reduces the original multi-frame design.
- Efficiency: GCNv2-SLAM runs at around 80 Hz on a laptop, while GCNv2-tiny runs at 40 Hz and its GCN-SLAM system runs at 20 Hz on the Jetson TX2.GCNv2-tiny halves the number of feature maps from conv2 onward.
- Efficiency: At equal resolution, GCNv2 has lower inference time than GCN and SuperPoint, while input-resolution changes affect inference time in a roughly quadratic manner.
B. Feature Extractor
GCNv2 produces keypoints and binary descriptors through a network designed for efficient matching and direct compatibility with ORB-based SLAM systems.
- GCNv2 returns a descriptor and keypoint confidence for an input image location using dense feature maps and bilinear sampling.
- Binarized features substantially accelerate matching relative to SuperPoint and GCN under the reported comparisons.
- A binary activation layer converts the final feature output into binarized descriptors while using a straight-through estimator for backpropagation.
- The 256-bit binary descriptor matches ORB’s size, enabling direct incorporation into ORB-SLAM2 and other ORB-based visual tracking systems.
- Descriptor training uses pixel-wise metric learning with squared Hamming distance, matching pairs from ground-truth poses, and mined non-matching pairs.
- The architecture includes exhaustive negative sample mining and relaxed criteria to increase tolerance to potentially noisy data.
C. Distributed Keypoint Detector
The detector predicts a keypoint probability map as a binary classification output and trains it across consecutive frames to improve tracking consistency.
- The detector treats each pixel as either a keypoint or a non-keypoint through a binary probability-map prediction.
- Weighted cross-entropy trains detection over two consecutive frames to enhance the consistency of extracted keypoints.
- Ground-truth keypoints come from Shi-Tomasi corners on a 16 × 16 grid, warped to the next frame using the ground-truth motion transform.
D. Training Details
Training combines descriptor and detector objectives with specified weights, margins, mining criteria, optimization settings, and a fixed training schedule.
- The final loss weights descriptor and detector terms by 100 and 1, respectively, to balance their scales.
- Training uses triplet margin m = 1, relaxed mining criteria c = 8, and cross-entropy weights [α1, α2] = [0.1, 1.0].
- ADAM training starts at a learning rate of 10^-4, halves it every 40 epochs, and runs for 100 epochs.
IV. GCN-SLAM
GCN-SLAM integrates GCNv2 into ORB-SLAM2 by replacing feature extraction and matching components while retaining the system’s loop-closure and pose-graph functions.
- ORB-SLAM2 uses ORB features because they are relatively cheap to compute and provide compact descriptors for fast matching.
- The original ORB-SLAM2 extractor builds a scale pyramid, applies FAST detection across grid cells, and culls detections to typically 1000 keypoints.
- GCN-SLAM replaces ORB-SLAM2’s conventional keypoint extraction with a single forward pass that computes locations and descriptors simultaneously.
- GCN-SLAM disables constant-velocity tracking and uses keypoint-based reference-frame tracking through matching to the last-created keyframe.
- Loop closure and pose-graph optimization remain intact, while the bag-of-words vocabulary is regenerated for GCNv2 descriptors.
V. EXPERIMENTAL RESULTS
The experiments evaluate GCN-SLAM as a motion-estimation keypoint method, emphasizing computational efficiency, embedded deployment, and suitability for SLAM rather than replacement of ORB-SLAM2.
- Experimental Results: The evaluation measures a keypoint extraction method tailored to motion estimation, computationally efficient, and suitable for SLAM.The authors explicitly frame GCN-SLAM as a keypoint extraction method rather than an alternative to ORB-SLAM2.
- Experimental Results: Quantitative experiments use an Intel i7-7700HQ laptop with a mobile NVIDIA 1070, while qualitative experiments use a Jetson TX2 and Intel RealSense D435 on a custom drone.
A. Training Data
GCNv2 is trained on a large RGB-D dataset with locally refined pose estimates and evaluated against GCN and other systems using frame-to-frame and closed-loop tracking.
- Training Data: GCNv2 is trained on 44,624 SUN-3D RGB-D frames spanning typical indoor environments, with relative poses refined through SIFT features and bundle adjustment.The provided poses are globally accurate but can be misaligned at the frame level, motivating local refinement.
- Quantitative Results: The comparison includes the original GCN, GCNv2-large, and GCNv2-tiny, with tracking evaluated using Absolute Trajectory Error on selected TUM sequences.GCNv2-large uses a ResNet-18 backbone, while GCNv2-tiny is assessed as a smaller variant.
- Quantitative Results: GCNv2 performs close to GCN and notably better than SuperPoint, with GCNv2-large recovering performance on the detail-sensitive fr1 360 sequence.GCNv2-tiny is only slightly less accurate than GCNv2; lower-scale feature maps reduce performance on fr1 floor and fr1 360.
- Quantitative Results: GCN-SLAM tracks all closed-loop sequences with error comparable to GCN, while ORB-SLAM2 fails on two sequences.GCNv2 has less error than ORB-SLAM2 on the fast rotations of fr1 360, although ORB-SLAM2 performs well on the other sequences.
C. Qualitative Results
Qualitative tests examine GCN-SLAM under indoor, outdoor, handheld, and drone conditions, showing maintained tracking and improved drone position holding relative to the compared baselines.
- Qualitative Results: The qualitative datasets include corridor traversal, outdoor circular motion, a 180-degree drone turn, and a 360-degree kitchen flight.
- Qualitative Results: ORB fails during the corridor turn and almost immediately in the outdoor circular sequence, whereas GCN-SLAM maintains tracking across all four datasets.The qualitative results have no ground truth and therefore supplement rather than replace the quantitative evaluation.
- Qualitative Results: Using GCN-SLAM rather than optical flow as the positioning source produces better drone position holding with less noise in the latter trajectory.
- Qualitative Results: GCNv2 produces fewer features than ORB but a higher percentage of inliers and better-distributed features in the adapted SLAM pipeline.
VI. CONCLUSIONS
GCNv2 addresses GCN’s real-time SLAM limitations with a smaller, more efficient design that is adaptable to existing systems. The paper verifies its tracking robustness through GCN-SLAM experiments and identifies indoor deployment as the target scope.
- Trajectory comparisons use the same GCN-SLAM pipeline with GCNv2 or ORB features, while the absence of ground truth makes these results qualitative.
- Keypoint tracking comparisons report total detections and the fraction successfully used for local map tracking across the Corridor and Kitchen datasets.
- Mesh reconstruction uses GCN-SLAM output for TSDF integration, with loop closure disabled to demonstrate tracking accuracy alone.
- GCNv2 is a smaller, more efficient version of GCN that is readily adaptable to existing SLAM systems.
- GCNv2 was incorporated into GCN-SLAM and used onboard for drone positioning to verify its robustness and performance.
- Limitations: GCNv2 is intentionally trained for projective geometry rather than generic feature matching, and the study targets indoor settings without further investigating outdoor performance.