Source-linked AI summary
RIO: 3D Object Instance Re-Localization in Changing Indoor Environments
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, Matthias Nießner
TL;DR
The paper studies estimating 6DoF poses of objects across RGB-D scans of the same changing environment. It introduces 3RScan and a fully-convolutional multi-scale correspondence network, achieving 30.58% accuracy on the re-localization benchmark.
Problem
RIO addresses estimating corresponding 6DoF poses for objects across scans of the same environment taken at different times, where changing geometry complicates feature matching.
Method
The paper introduces 3RScan and a fully-convolutional multi-scale network that learns change-invariant geometric features from corresponding TSDF patches.
Results
The method outperforms state-of-the-art baselines on correspondence matching and the object-instance re-localization benchmark, reaching an F1 score of 94.37 with dynamic-data training.
Takeaways & Limitations
3RScan supports training and evaluation for object re-localization and other persistent scene-understanding tasks in changing indoor environments.
Takeaways & Limitations
Evaluation accounts for rotational symmetry because 22% of dataset objects are symmetric.
Abstract
from arXiv · showhide
In this work, we introduce the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time. We consider RIO a particularly important task in 3D vision since it enables a wide range of practical applications, including AI-assistants or robots that are asked to find a specific object in a 3D scene. To address this problem, we first introduce 3RScan, a novel dataset and benchmark, which features 1482 RGB-D scans of 478 environments across multiple time steps. Each scene includes several objects whose positions change over time, together with ground truth annotations of object instances and their respective 6DoF mappings among re-scans. Automatically finding 6DoF object poses leads to a particular challenging feature matching task due to varying partial observations and changes in the surrounding context. To this end, we introduce a new data-driven approach that efficiently finds matching features using a fully-convolutional 3D correspondence network operating on multiple spatial scales. Combined with a 6DoF pose optimization, our method outperforms state-of-the-art baselines on our newly-established benchmark, achieving an accuracy of 30.58%.
1 Technical University of Munich 2 Google
The paper introduces a benchmark for estimating 6DoF poses of changed rigid object instances between segmented source and later target scans.
- The benchmark estimates 6DoF poses for changed rigid object instances from a segmented source scan to a later target scan.
1. Introduction
The paper defines object instance re-localization in changing indoor environments and addresses the lack of temporal supervision with 3RScan and a multi-scale feature-learning approach.
- RIO estimates corresponding 6DoF object poses between RGB-D scans of the same environment acquired at different times.
- Changing scanning patterns and geometric context make correspondence establishment difficult and reduce the effectiveness of hand-crafted descriptors.
- The dataset includes calibrated RGB-D sequences, aligned textured reconstructions, dense instance-level segmentation, and symmetry-aware local alignment of changes.
- 3RScan provides 1482 RGB-D sequences with repeated environment scans, dense instance annotations, and associations for objects changed across re-scans.
- The proposed fully-convolutional multi-scale network learns change-invariant geometric features from corresponding TSDF patches at two spatial scales.
2. Related Work
Prior RGB-D scene-understanding research largely assumes static environments, while existing datasets provide limited real-world temporal scene-change coverage.
- 3D Object Localization and Keypoint Matching: 3D localization research has used both hand-crafted descriptors and learned feature descriptors for point-to-point correspondence matching.
- RGB-D Scene Understanding: RGB-D scene-understanding methods reconstruct geometry and fuse semantic information using volumetric or surfel-based SLAM, object-level systems, or direct scan processing.
- RGB-D Datasets: Most RGB-D datasets assume static scenes and omit changes in scene layout and object placement, although ScanNet contains 1500 scans of approximately 750 unique scenes.
3. 3RScan-Dataset
3RScan is a real-world dataset of repeatedly scanned, naturally changing indoor environments, with annotations supporting instance-level re-localization and 6DoF object alignment.
- Dataset overview: 3RScan contains multiple 3D snapshots of naturally changing indoor environments for benchmarking tasks such as long-term SLAM.Scenes have 2–12 re-scans, with controlled and uncontrolled temporal changes.
- Dataset overview: 1482 sequences were recorded using a Tango mobile application and reconstructed offline with bundle-adjusted camera poses, loop closure, and texture mapping.
- Annotations: The dataset provides calibrated RGB-D sequences, camera poses, calibration parameters, global scan alignment, and dense instance-level semantic segmentation.Instance IDs remain consistent across sequences from the same environment.
- Annotations: Each changed object has a ground-truth 3D transformation and symmetry property, while ambiguous instances include intra-class transformations for all valid poses.
- Changing environments: Scene changes include rigid object motion, object removal or addition, non-rigid objects, and lighting changes.Moved objects can shift from a few centimeters to a few meters.
- Annotation process: Instance annotations are propagated from reference scans using global alignment, then object transformations are verified through manually annotated keypoints and Procrustes alignment.
4. 3D Object Instance Re-Localization
RIO re-localizes source object instances in later scans by learning multi-scale, change-invariant 3D correspondences and optimizing a rigid 6DoF alignment.
- Network architecture: The correspondence network processes TSDF patches at two spatial scales to encode change-invariant neighborhood information around objects and scenes.The patches use 32 × 32 × 32 voxel grids covering (1.2 m)^3 and (0.6 m)^3 volumes.
- Network architecture: Two single-scale encoder branches feed a multi-scale encoder, combining fine geometric details with higher-level contextual semantics.The branches produce a 512-dimensional feature vector after non-padded convolutions and pooling.
- Training: Training uses triplets that minimize positive-patch distances and maximize negative-patch distances with margin α = 1.Static self-supervised training is followed by fine-tuning on dynamic patches around moving objects.
- Training: Dynamic correspondence pairs use ground-truth pose annotations, negatives from other scenes or removed objects, and random rotation augmentation.Fine-tuning freezes the first layers and trains the multi-scale encoder branch.
- Re-localization pipeline: The method computes source-object and target-scene features, retrieves correspondences in latent space, filters outliers with RANSAC, and estimates a 6DoF pose.
- 6DoF pose alignment: The alignment step finds a rigid transformation between corresponding source-object and target-scene points, solved using Singular Value Decomposition.
5. Evaluation
The evaluation measures dynamic correspondence matching and 3D object instance re-localization on 3RScan. The proposed multi-scale method outperforms the evaluated single-scale, hand-crafted, and learned baselines.
- Evaluation setup: The evaluation first compares dynamic keypoint matching and then assesses object instance re-localization on the 3RScan benchmark.The correspondence experiment uses annotated changed objects, while the benchmark evaluates predicted object transformations.
- 5.1. Correspondence Matching: At 95% recall, the multi-scale network trained with dynamic data reaches an F1 score of 94.37.The same architecture trained only with static data already outperforms all evaluated single-scale architectures by a large margin.
- 5.1. Correspondence Matching: The top-1 metric measures the percentage of top-ranked positive matches among 50 randomly selected negative patches.This metric is intended to represent re-localization cases in which multiple negative samples compete with a positive keypoint.
- 5.2. Object Instance Re-localization: Object poses are estimated by sampling keypoints, extracting descriptors, matching correspondences, and filtering them with RANSAC.The predicted rotation and translation are evaluated against ground truth using translation and rotation error thresholds.
- 5.2. Object Instance Re-localization: An instance is counted as successfully aligned when translation and rotation errors are below either 10 cm and 10° or 20 cm and 20°.Symmetries are considered during error computation.
- 5.2. Object Instance Re-localization: The proposed method outperforms classical hand-crafted methods by a large margin, although those methods remain reasonable for descriptive objects such as sofas and beds.Qualitative examples are provided for re-localization in changing environments.
6. Conclusion
3RScan provides large-scale, temporally discontinuous scans of real-world environments with annotations supporting object alignment and several benchmark tasks. The authors position RIO as a challenging task for persistent, long-term indoor-scene understanding.
- 6. Conclusion: 3RScan is described as the first large-scale dataset of real-world sequences with temporal discontinuity and multiple scans of the same environment.It includes 6D object instance alignments in changing environments.
- 6. Conclusion: The dataset includes annotations for persistent dense and sparse SLAM, change detection, camera re-localization, and other benchmark tasks.The authors intend 3RScan to support development and evaluation of algorithms for persistent indoor-scene understanding.
Dataset
3RScan combines repeated RGB-D scans of indoor environments with extensive instance, alignment, semantic, and scene-functionality annotations. Its acquisition and annotation pipeline supports studying object changes and multiple 3D-scene tasks.
- Acquisition: The scanning interface used a Google Tango mobile app with pre-annotation functionality and lightweight user guidance about scene changes.The interface is illustrated in Figure 9.
- Scene Matching and Alignment: Scene candidates were found through 2D texture-map similarity search before manual adjustment and coarse-to-fine global registration.Registration combines correspondence-based 2D ICP, RANSAC, and global 3D ICP.
- Preprocessing: The dataset has average scene instance-segmentation coverage above 98%, with 48k instances and 534 unique labels.Almost all scans have instance-segmentation coverage above 90%.
- RGB-D sequences: The dataset contains around 363k calibrated RGB-D and depth images from spatially and temporally calibrated Tango sequences.Median filtering removes rectification lines from Google Tango depth images before calibration.
- Scene Type: Scenes receive multiple functionality labels, including sleeping, eating, and working, rather than a single room-type label.This n-to-m labeling reflects the variety of scenes in 3RScan.
- Instance Change Annotation: The instance-change tool renders a scene beside its reference, propagates selected reference instances, and supports corresponding-keypoint selection.A Kabsch-based Procrustes alignment computes the transformation once enough keypoints are selected.
- Symmetry: Twenty-two percent of changed objects are annotated with rotational symmetry classes C2, C4, or C∞, and symmetry is included during pose evaluation.The symmetry properties are summarized in Table 6.
- Object Changes: Changed objects span 187 categories, dominated by movable furniture and portable items, while some appearance changes such as toilets are also annotated.Examples include chairs, pillows, boxes, smaller tables, sofas, beds, appliances, and cushions.