Source-linked AI summary
On-the-Fly Adaptation of Regression Forests for Online Camera Relocalisation
Tommaso Cavallari, Stuart Golodetz, Nicholas A. Lord, Julien Valentin, Luigi Di Stefano, Philip H. S. Torr
TL;DR
Camera relocalisation helps recover pose after tracking loss, but regression forests still require offline training on the target scene. This paper adapts a generic-scene forest online, achieving performance competitive with offline forests on 7-Scenes in under 150ms.
Problem
Method
The method adapts a generic-scene pretrained regression forest online by retaining its branching structure and refilling its leaves with new-scene training examples.
Source: §2.2.2 · p. 4Results
Under 150ms, the approach achieves relocalisation performance on 7-Scenes competitive with offline-trained forests and can relocalise from novel poses.
Source: §4 · p. 8Takeaways & Limitations
Compared with fast keyframe-based approaches, the method is better able to relocalise from novel poses, freeing users from manually searching for known poses.
Source: §4 · p. 8The practicality trade-off includes a performance drop, though ICP-based pose refinement increases the proportion of test frames relocalised.
Source: §3.1 · p. 6
Abstract
from arXiv · showhide
Camera relocalisation is an important problem in computer vision, with applications in simultaneous localisation and mapping, virtual/augmented reality and navigation. Common techniques either match the current image against keyframes with known poses coming from a tracker, or establish 2D-to-3D correspondences between keypoints in the current image and points in the scene in order to estimate the camera pose. Recently, regression forests have become a popular alternative to establish such correspondences. They achieve accurate results, but must be trained offline on the target scene, preventing relocalisation in new environments. In this paper, we show how to circumvent this limitation by adapting a pre-trained forest to a new scene on the fly. Our adapted forests achieve relocalisation performance that is on par with that of offline forests, and our approach runs in under 150ms, making it desirable for real-time systems that require online relocalisation.
1. Introduction
Relocalisation helps recover camera pose after tracking loss and supports loop closure. This paper addresses the need for scene-specific offline training by adapting regression forests to new scenes online.
1. Introduction
- Relocalisation lets SLAM recover after tracking loss rather than restart reconstruction, and is also important for loop closure when building globally consistent maps.Source: §1 · p. 1
- Traditional methods match images to stored keyframes or estimate pose from 2D-to-3D keypoint correspondences; hybrid methods combine retrieval with geometric refinement.Source: §1 · pp. 1–2
- Regression forests predict 3D correspondences without explicit keypoint detection, description, or matching, but require offline training on the target scene.Prior work improved their relocalisation performance, yet did not remove the need for target-scene training.Source: §1 · pp. 2–3
- Online adaptation retains learned tree branches while replacing scene-specific leaf information, yielding relocalisation competitive with offline forests and supporting novel poses.The adapted forest predicts correspondences for pose estimation.Source: Figure 1 · p. 2, §1 · p. 3
2. Method
The method reuses a generic-scene regression forest’s branching structure while learning new-scene leaf distributions online. Those distributions provide scene correspondences for camera-pose estimation.
2.1. Overview
- The approach retains a generic-scene forest’s tree structure, replaces its leaf contents, and refills leaves online with distributions specific to the new scene.The adapted forest then predicts correspondences for camera-pose estimation.Source: §2.1 · p. 3
2.2.1 Offline Forest Training
- Training examples use 256D depth and DA-RGB pixel features sampled from tracked RGB-D video with known camera poses.The 256 dimensions concatenate 128 randomly generated Depth and 128 DA-RGB feature evaluations.Source: §2.2.1 · p. 3
- Each tree recursively splits examples using one of 512 candidate thresholds selected to maximize spatial-variance reduction; trees grow to depth 15, with five trees per forest.The learned leaf distributions are discarded after pre-training for replacement during online adaptation.Source: §2.2.1 · p. 3
2.2.2 Online Forest Adaptation
- Online adaptation adds new-scene 3D points and colors to per-leaf reservoirs from reliable-tracker frames, then clusters them into modes while updating 256 leaves per frame.A typical forest has about 42,000 leaves, so each is updated roughly once every six seconds.Source: §2.2.2 · p. 4
2.2.3 Camera Pose Estimation
- Pose estimation scores hypotheses by Mahalanobis distance to forest modes, then prunes and refines them with pre-emptive RANSAC and Levenberg–Marquardt; ICP refinement is optional.The forest preserves effective routing to leaves even as adaptation changes the predicted mode locations.Source: §2.2.2 · p. 4, §2.2.3 · p. 5
3. Experiments
Experiments compare adaptation with offline methods, test live-sequence recovery and novel-pose generalisation, and assess runtime against random ferns. Adapted forests approach offline performance, recover quickly, and generalise to poses farther from the training trajectory, with slower but acceptable timings.
3.1. Adaptation Performance
- Adapted forests come within striking distance of offline methods and outperform online random ferns; Table 2 reports competitive or sometimes better results without target-scene pre-training.ICP refinement raises the proportion of test frames successfully relocalised while adding little runtime, keeping the approach practical for online use.Source: §3.1 · p. 6, Table 2 · p. 7
3.2. Tracking Loss Recovery
- Relocalisation starts typically after 4–6 frames of live-sequence adaptation, and subsequent performance rarely falls below 80% except on Stairs.The procedure tests each new frame for recovery and continues training from that frame when tracking has not actually failed.Source: §3.2 · p. 7, Figure 3 · p. 7
3.3. Generalisation to Novel Poses
- Performance decays less sharply than with random ferns as test poses move farther from the training trajectory; on Fire, successful relocalisations include poses over 35cm/35° away.The Fire examples meet the stated 5cm/5° relocalisation threshold, including the hard-pose category.Source: Figure 4 · p. 7, §3.3 · p. 7, Figure 5 · p. 8
3.4. Timings
- Our approach takes 9.8 ms for online training and 141 ms for relocalisation, versus random ferns’ 0.9 ms per frame and 10 ms, while covering a broader pose range.The authors describe these timings as acceptable and note that their research-focused implementation is not heavily optimised.Source: §3.4 · p. 8, Table 3 · p. 8
4. Conclusion
Offline regression methods require advance training on the target scene, limiting their use for online relocalisation. Adapting a pre-trained forest online achieves performance competitive with offline forests, runs in under 150 ms, and supports relocalisation from novel poses.
4. Conclusion
- Offline regression-based correspondence methods achieve state-of-the-art relocalisation, but require extensive training on the target scene in advance.This has hindered their adoption for online relocalisation in practical systems such as InfiniTAM.Source: §4 · p. 8
- On 7-Scenes, adapted forests achieve relocalisation performance competitive with offline-trained forests and run in under 150 ms, making the approach practically competitive with fast keyframe methods such as random ferns.Compared with those approaches, adapted forests are much better able to relocalise from novel poses.Source: §4 · p. 8
- The paper adapts regression forests trained offline on another scene to novel scenes online.This approach circumvents the need for offline training on the scene of interest.Source: §4 · p. 8
A. Analysis of Failure Cases
The section examines two failure cases to explain situations in which the approach can fail.
A. Analysis of Failure Cases
- The approach can still fail in some situations, and the section examines two cases to explain why.Source: A. Analysis of Failure Cases · p. 11
A.1. Office
In the Office scene, visually similar desks can confuse the relocaliser, producing a pose facing the wrong desk. In the analysed case, several successful candidates remain, but RANSAC selects the wrong one because correct and incorrect hypotheses have similar energies.
A.1. Office
- Visually similar desks in the Office scene can lead the relocaliser to choose a pose facing the wrong desk.The desks share similar materials and furnishings.Source: §A.1 · p. 11, Figure 6 · p. 11
- The wrong-desk pose looks superficially plausible and is oriented at roughly the right angle, but is far from ground truth.Source: §A.1 · p. 11
- The top RANSAC candidate faces the wrong desk, although any of the next five candidates would relocalise correctly.The energies of correct and incorrect hypotheses are fairly similar.Source: §A.1 · p. 11, Figure 7 · p. 12
- Photometric consistency between colour raycasts from surviving poses and the input image is suggested as a possible scoring signal, but is not investigated.Source: §A.1 · p. 11
A.2. Stairs
In the Stairs scene, visually identical steps make relocalisation prone to failure from viewpoints where the system must rely on ambiguous points. Similar leaves for corresponding points on different stairs make their distributions less informative and reduce the chance of generating useful hypotheses.
A.2. Stairs
- Visually identical stairs make relocalisation prone to fail from viewpoints where unambiguous scene points are unavailable.From some angles, identifiable points do allow the relocaliser to estimate the pose correctly.Source: §A.2 · p. 13, Figure 9 · p. 14
- Points at corresponding locations on different stairs tend to fall into similar leaves, making leaf modes less informative and reducing the probability of good initial hypotheses.The predicted points can be roughly in the right place but on the wrong stairs.Source: §A.2 · p. 13, Figure 10 · p. 14
- None of the top 16 hypotheses in the analysed Stairs failure case would have relocalised successfully.Source: §A.2 · p. 13
- A late-stage consistency check cannot fix this case because none of the surviving hypotheses is useful.Possible alternatives are to diversify initial hypotheses or increase their number in difficult conditions.Source: §A.2 · p. 13
B. Further Successful Examples
In the Fire scene, relocalisation examples show the system relying on unambiguous scene points, which are easier to find among easily distinguished objects. By contrast, none of the top 16 Stairs pose candidates succeeds, likely because similar stair locations produce less informative leaf modes.
B. Further Successful Examples
- Fire examples show the relocaliser relying on unambiguous points, which are easier to find among easily distinguished objects.Figure 11 provides further successful relocalisation examples from the Fire scene.Source: B. Further Successful Examples · p. 14, Figure 11 · p. 14
- None of the top 16 pose candidates for the Stairs failure case would relocalise the camera successfully.The coloured points mark the 2D-to-3D correspondences used to generate the initial pose hypotheses.Source: Figure 12 · p. 15
- Similar points on different stairs tend to reach similar leaves, making leaf modes less informative and reducing the probability of good initial hypotheses.Source: Figure 12 · p. 15