RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

Introduction

• RIO estimates one or more objects’ 6DoF poses in another scan of the same environment captured at a different time.The task targets changing indoor environments, with applications such as helping a robot or virtual assistant find a specific object.Source: §1 · p. 1
• Correspondence matching is difficult because scans have different patterns and changing geometric context, making hand-crafted and static-scene learned features less effective.Existing RGB-D datasets lack temporal annotations across scene changes, limiting supervised training data for object instance localization.Source: §1 · pp. 1–2
• 3RScan comprises 1482 RGB-D sequences with multiple re-scans per environment, dense instance annotations, and associations for objects changed in appearance or location.The dataset is also introduced as a benchmark for object instance localization.Source: §1 · p. 2
• A fully-convolutional multi-scale network learns geometric features for changing environments using corresponding TSDF patches on moved objects at two spatial scales.The approach learns change-invariant local features and is reported to outperform state-of-the-art baselines in correspondence matching and object instance re-localization.Source: §1 · p. 2

3D Object Localization and Keypoint Matching

• Learned 3D descriptors support matching and registration but are restricted to static environments, whereas this work targets object localization in cluttered scans with geometric and appearance changes.Earlier methods used handcrafted descriptors for point correspondences under occlusion and clutter; interest later shifted to learned descriptors [32] [19] [7].Source: §2 · p. 2

RGB-D Scene Understanding

• RGB-D scene-understanding approaches use SLAM-based reconstruction or direct scan segmentation, but assume objects remain static and scene structure does not change over time.SLAM++ [24] and Fusion++ [12] operate at the object level to build semantic scene graphs for SLAM and loop closure.Source: §2 · pp. 2–3

RGB-D Datasets

• Most RGB-D datasets assume static scenes, while only a few proposals capture changes; the cited real dataset is too small and lacks annotations for training and testing learned approaches.ScanNet [4] has 1500 scans of approximately 750 unique scenes. InteriorNet [11] is synthetic and applies furniture shuffles and illumination changes; Fehr et al. released a real dataset of 23 sequences from 3 rooms, used for segmenting scene structure from movable furniture.Source: §2 · p. 3

3. 3RScan-Dataset

• 3RScan contains 2–12 snapshots of naturally changing indoor environments and is intended for tasks including long-term SLAM, scene-change detection, and camera or object-instance re-localization.Source: §3 · p. 3

3.1. Overview

• The dataset comprises 1,482 sequences recorded with a Tango mobile application across more than 13 countries by 45+ people, with approximately 363,000 annotated 2D frames and aligned 3D data.Sequences include bundle-adjusted camera poses, loop closure, and texture-mapped 3D reconstructions.Source: §3.1 · p. 3

3.2. Scene Changes

• Recorded scene changes include rigid objects moved by centimeters to meters or added or removed, as well as non-rigid changes and lighting variation.Source: §3.2 · p. 4

3.3. Annotation

• The annotations include scan-to-scene mappings, dense instance segmentation, and 3D transformations for changed objects.The transformations include object alignment and symmetry properties.Source: §3.1 · p. 3, §3.3 · p. 4

3.3.1 Semantic Segmentation

• Semantic labels are drawn on segmented 3D surfaces in reference scans, then propagated to re-scans using global alignment under the assumption that most of the scene remains static.Human experts annotate and verify segments; average semantic-segmentation coverage is 98.5%.Source: §3.3.1 · p. 4

3.3.2 Instance Changes

• Instance-level 3D transformations are computed by applying Procrustes to manually annotated object keypoint correspondences, with a symmetry property assigned to each instance.Source: §3.3.2 · p. 4

3.4. Benchmark

• The benchmark evaluates 6DoF object-instance pose estimation from a segmented source scene to a later scan, using translation and rotation thresholds of t ≤10 cm and r ≤10° or t ≤20 cm and r ≤20°.Symmetry is considered; standardized training, validation, and test sets and a hidden test set with server-side testing are provided.Source: §3.4 · p. 4

4.1. Data Representation

• Each keypoint is represented by TSDF patches on two 32 × 32 × 32 voxel grids, spanning spatial resolutions of (1.2 m)3 and (0.6 m)3.The passage also gives voxel sizes of 1.875 cm and then cuts off before specifying the second size.Source: §4.1 · p. 4

4.2. Network Architecture

• The network concatenates separate single-scale encoders into a multi-scale encoder, capturing fine geometry and broader context in a 512-dimensional feature that outperforms independently processed single scales.The two input resolutions provide neighborhoods at different levels of detail.Source: §4.2 · p. 5

4.3. Training

• Training uses a triplet loss to bring positive patch pairs together and separate negative pairs, with margin α = 1 and Adam initialized at learning rate 0.001.Source: §4.3 · p. 5

4.4. Training Data: From Static to Dynamic

• Static self-supervised TSDF patch pretraining is followed by dynamic fine-tuning on moving-object patches, freezing early layers and training the multi-scale encoder.Static patch pairs come from non-overlapping frame sets and corresponding Harris 3D keypoints in two TSDF volumes. Dynamic correspondence pairs use pose annotations; negatives include patches from other scenes and removed objects, with random rotations augmenting training.Source: §4.4 · pp. 5–6

4.5. 6DoF Pose Alignment

• Features for source-object and target-scene keypoints are matched by latent-space k-nearest neighbors, filtered with RANSAC, and aligned through SVD-based 6DoF pose optimization.The optimization finds a rigid rotation and translation from the remaining correspondence sets.Source: §4.5 · p. 6

5.1. Correspondence Matching

• Matching is assessed with F1, accuracy, precision, FPR, and ER at 95% recall, plus top-1 placement accuracy against 50 randomly chosen negative patches.Source: §5.1 · pp. 6–7
• 94.37 F1 is achieved by the multiscale network after dynamic-data fine-tuning; even when trained only on static data, it outperforms all single-scale architectures by a large margin.Source: §5.1 · p. 7, Table 3 · p. 7

5.2. Object Instance Re-localization

• RIO evaluation compares the method with PCL hand-crafted SHOT and FPFH features and learned 3DMatch, computing each instance’s transform through keypoint sampling, descriptors, matching, and RANSAC filtering.Source: §5.2 · p. 7
• Successful alignment requires translation and rotation errors below either 10 cm and 10° or 20 cm and 20°, with object symmetries considered in error computation.Source: §5.2 · p. 8
• The proposed method outperforms classical hand-crafted features by a large margin across object instances, though those features remain reasonably effective for descriptive sofas and beds.Results are reported for all instances and by category.Source: §5.2 · p. 8, Table 4 · p. 8, Table 5 · p. 8

• 3RScan is the first large-scale dataset of real-world sequences with temporal discontinuity, comprising multiple scans of the same environment.Source: §6 · p. 8
• RIO in changing indoor environments is a challenging and important task that remains to be further explored.Source: §6 · p. 8
• Beyond 6D object-instance alignments, 3RScan provides annotations for tasks including persistent dense and sparse SLAM, change detection, and camera re-localization.The authors expect the dataset to help develop and evaluate algorithms toward persistent, long-term understanding of indoor environments.Source: §6 · p. 8

Scanning Interface

• A mobile Google Tango app with pre-annotation functionality served as the scanning interface, and some users supplied lightweight scene-change instructions for later annotation.Figure 9 shows the annotation interface used for data acquisition.Source: Dataset · p. 11, Figure 9 · p. 11

Scene Matching and Alignment

• Scene candidates are found through 2D texture-UV similarity search, then manually adjusted and globally registered with coarse-to-fine 2D ICP and RANSAC followed by 3D ICP.Additional verification and optional manual keypoint-based alignment support alignment quality.Source: Dataset · p. 11

Preprocessing

• Offline 3D segmentation and scene alignment support propagation of semantic labels from references to re-scans, followed by manual cleanup to produce instance segmentations.The semantic segmentation interface is attributed to Dai et al. [4].Source: Dataset · p. 11
• 48k instances are annotated with 534 unique labels, and almost all scans have instance segmentation coverage above 90%, with average scene coverage above 98%.Figure 10 summarizes the number of scans by instance-annotation coverage.Source: Dataset · p. 11, Figure 10 · p. 11

RGB-D sequences

• The dataset contains around 363k calibrated RGB-D and depth images, with spatial and temporal calibration and median filtering to remove rectification lines from Tango depth images.The RGB-D sequences are processed offline into texture-mapped 3D reconstructions.Source: Dataset · p. 11

Scene Type

• Scenes are annotated with multiple corresponding functionalities, such as sleeping, eating, and working, rather than a one-to-one assignment of room types.The passage describes this as an n-to-m annotation scheme and notes the variety of scenes in 3RScan.Source: Dataset · p. 11

Instance Change Annotation

• A web-based tool aligns each re-scan object to its reference using selected keypoints and Kabsch-based Procrustes alignment, while tracking IDs for non-rigid, added, and removed objects.Selecting a re-scan object automatically segments its reference instance, requiring consistent instance IDs across scans of the same environment.Source: Dataset · pp. 11–12

Symmetry

• 22% of changed objects are symmetric, categorized by rotational symmetry around the canonical axis as C2, C4, or C∞, and symmetry is considered when evaluating predicted poses.The symmetry annotation follows Avetisyan et al. [2].Source: Dataset · p. 12