Source-linked AI summary
ORBSLAM-based Endoscope Tracking and 3D Reconstruction
Nader Mahmoud, Iñigo Cirauqui, Alexandre Hostettler, Christophe Doignon, Luc Soler, Jacques Marescaux, J. M. M. Montiel
TL;DR
Real-time monocular endoscope tracking and 3D reconstruction could support surgical navigation and augmented reality, but existing monocular structure-from-motion methods require offline processing and ORBSLAM maps poorly represent textureless soft organs. The authors retune ORBSLAM for monocular endoscope tracking and add a matching stage to reconstruct a semi-dense map of soft organs. The system achieved 3-4.1 mm RMSE against a CT ground-truth surface and showed robust performance in in-vivo sequences during partial occlusions and severe deformations.
Problem
Real-time monocular endoscope tracking and 3D reconstruction could support surgical navigation and augmented reality, but existing monocular structure-from-motion methods require offline processing and ORBSLAM maps poorly represent textureless soft organs.
Source: §1 · pp. 1–2Method
The authors retune ORBSLAM for monocular endoscope tracking and add a matching stage to reconstruct a semi-dense map of soft organs.
Source: §1 · p. 2Results
The system achieved 3-4.1 mm RMSE against a CT ground-truth surface and showed robust performance in in-vivo sequences during partial occlusions and severe deformations.
Source: §5 · p. 11Takeaways & Limitations
The image-stream-only system tracked the endoscope and reconstructed a semi-dense scene map in the tested in-vivo sequences.
Source: §5 · p. 11Only 24% of visible map points were matched, with ORB’s limited repeatability on soft organs identified as a main reason for missed matches.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
We aim to track the endoscope location inside the surgical scene and provide 3D reconstruction, in real-time, from the sole input of the image sequence captured by the monocular endoscope. This information offers new possibilities for developing surgical navigation and augmented reality applications. The main benefit of this approach is the lack of extra tracking elements which can disturb the surgeon performance in the clinical routine. It is our first contribution to exploit ORBSLAM, one of the best performing monocular SLAM algorithms, to estimate both of the endoscope location, and 3D structure of the surgical scene. However, the reconstructed 3D map poorly describe textureless soft organ surfaces such as liver. It is our second contribution to extend ORBSLAM to be able to reconstruct a semi-dense map of soft organs. Experimental results on in-vivo pigs, shows a robust endoscope tracking even with organs deformations and partial instrument occlusions. It also shows the reconstruction density, and accuracy against ground truth surface obtained from CT.
1 Introduction
Real-time monocular endoscope tracking and 3D reconstruction could support surgical navigation and augmented-reality overlays in minimally invasive surgery. Existing reconstruction and tracking approaches described here either use stereo endoscopes or require offline processing, motivating monocular real-time SLAM.
1 Introduction
- Monocular real-time reconstruction alongside endoscope tracking provides information for surgical navigation and augmented-reality overlays of preoperative data.These applications address limited field of view and lack of depth perception in minimally invasive surgery.Source: §1 · p. 1
- Stereo-endoscope computer-vision methods provide intra-operative reconstruction and tracking but are not adapted to commonly used monocular endoscopes.The cited stereo methods are.Source: §1 · pp. 1–2
- Structure-from-motion methods address monocular endoscopes, but their offline batch processing is unsuitable for real-time applications.In, a tracking sensor is attached to estimate endoscope position.Source: §1 · p. 2
- MIS VSLAM research includes stereo-endoscope EKF-SLAM and a method that learns periodic liver-motion parameters to improve estimation.The cited work applies EKF-SLAM to MIS and models periodic liver deformation.Source: §1 · p. 2
- Monocular endoscopy remains challenging in non-rigid surgical scenes, although prior work demonstrated monocular EKF-SLAM feasibility and validated it on in-vivo human sequences.The latter work showed use for hernia-defect measurements in hernia repair.Source: §1 · p. 2
- The paper retunes ORBSLAM for MIS and adds a matching algorithm to densify its map, reporting robust tracking and 3–4.5 mm accuracy against a CT ground-truth surface.The authors also report successful relocation after tracking loss and AR overlays of preoperative models.Source: §1 · p. 2
2 ORBSLAM overview
ORBSLAM uses keyframes, feature matching, optimization, and place recognition to track the endoscope and build a 3D map. Mapping and tracking proceed in parallel, while relocation uses stored keyframes to recover a pose after tracking loss.
2 ORBSLAM overview
- ORBSLAM combines keyframes and nonlinear optimization with a covisibility graph, DBoW2 place recognition, and scale-aware loop closing.ORB features serve mapping and place recognition.Source: §2 · pp. 2, 4
- Tracking predicts pose from the previous frame, matches reprojected map points to nearby ORB features, and refines pose by robust nonlinear optimization of reprojection error.Matches are classified as inliers or outliers, and consistently unreliable points are discarded during initialization.Source: §2 · p. 3
- Mapping selects keyframes, matches their features, and computes keyframe poses and 3D map points by bundle adjustment in a lower-frequency thread parallel to tracking.Bundle adjustment minimizes robustified reprojection error; new keyframes initialize additional map points from matches to previous keyframes.Source: §2 · p. 3
- Map-point culling removes points that fail subsequent tracking, have low triangulation parallax, or produce excessive reprojection error; redundant keyframes are also removed.A keyframe is removed when 90% of its map points are detected in at least three other keyframes.Source: §2 · pp. 3–4
- After tracking loss, ORB features query a bag-of-words keyframe database and covisibility graph; P3P with RANSAC estimates a valid pose so tracking can resume.Tracking loss may follow occlusion, fast motion, or too few map-point matches.Source: §2 · p. 4
3 Extending the map density
The map extension targets soft-organ regions where ORB features are not repeatedly detected across frames. It adds epipolar-guided patch matching for unmatched features and triangulates the resulting correspondences into a semi-dense map.
3 Extending the map density
- The extension uses epipolar-guided cross-correlation to match unmatched ORB features in a new keyframe, enabling triangulation where repeatable ORB detections are lacking on soft organs.The standard mapping process triangulates matched ORB points, but cannot initialize points on soft organs such as liver when features are not repeatedly detected.Source: §3 · p. 4
- The map adds reconstructed points in sparse image regions, shown as green points alongside the original red ORBSLAM map; additional points appear as new regions are explored.The figure also shows reprojections and two views of the maps.Source: §3 · p. 4, Figure 1 · p. 5
- For each new keyframe, Algorithm 1 searches four neighboring keyframes with significant baseline and matches unmatched features by patch cross-correlation around epipolar segments.It rejects matches too far from the epipolar line, then triangulates the matched observations.Source: §3 · p. 5
- Algorithm 1 filters triangulated points by negative depth, reprojection error, and disagreement with the median depth in the current keyframe.The resulting points are tracked in later frames first by Lucas–Kanade optical flow, then by epipolar-guided cross-correlation for untracked points.Source: §3 · p. 5
4 Experimental results
In-vivo experiments evaluated retuned ORBSLAM tracking and reconstruction, then tested semi-dense mapping against CT and under deformation and partial occlusion. The system tracked through these challenges and reconstructed maps with approximately 3–4.1 mm RMSE against CT.
4.1 ORBSLAM performance evaluation
- Despite matching only 24% of visible map points, ORBSLAM produced a sparse map of 66 keyframes and 1,566 points and estimated breathing motion.The endoscope was fixed relative to the operating table during the breathing-motion observation.Source: §4.1 · p. 6, Figure 2 · p. 7
- After extraction caused tracking loss while viewing the liver, the system relocated within 3 seconds when reinserted and imaging previously mapped spleen points.The relocation sequence is shown in Fig. 3.Source: §4.1 · p. 6, Figure 3 · p. 7
- The system was tested on challenging gastroscopy sequences with reflections and abrupt movements, tracking endoscope location and reconstructing a 3D scene map at approximately 25 ms average tracking time per frame.Timing was reported on a desktop PC with 6 GB RAM.Source: §4.1 · p. 7
4.2 Estimation of reconstruction error
- CT comparison used in-vivo pig sequences recorded before liver interaction, followed by CT acquisition with the endoscope fixed and its tip inside the abdominal cavity.The abdominal surface was segmented as ground truth; scale and roll alignment were needed before map comparison.Source: §4.2 · pp. 8–9, Figure 5 · p. 8
- Approximately 3 mm RMSE for ORB points and 4.1 mm for the semi-dense map were measured against CT surface ground truth.The comparison used the closest CT surface points; 80% of map points were included, excluding outliers and points outside the CT acquisition field.Source: §4.2 · p. 9, Figure 6 · p. 9
4.3 Evaluation with instrument occlusion and deformations
- Tracking remained successful during liver interaction across sequences with deformations and partial scene occlusion, including a textureless liver, where many points were reconstructed.When fast endoscope movement caused tracking failure through feature deletion, the system could relocate after movement and detection of a few ORB features.Source: §4.3 · pp. 9–10, Figure 8 · p. 10
5 Conclusion and Future Work
The paper retunes ORBSLAM for monocular endoscope tracking and 3D reconstruction from the endoscope image stream alone. It reports robust in-vivo performance and identifies non-rigid organ deformation estimation as future work.
5 Conclusion and Future Work
- Retuned ORBSLAM tracks the monocular endoscope and reconstructs the 3D scene using only the endoscope image stream.Source: §5 · p. 11
- The system showed robust performance despite partial occlusions and severe deformations, with 3–4.1 mm RMSE against CT ground truth.Source: §5 · p. 11
- Future work will use the semi-dense map and tracked 2D image points to estimate non-rigid organ deformation with shape-from-template techniques.Source: §5 · p. 11