Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion

1 Introduction

• The method addresses a central fusion challenge: object counts are not fixed, and detections from different views receive inconsistent identifiers.Densely labeled 3D datasets are challenging to acquire, while annotations and pre-trained predictors for 2D data are widely available.Source: §1 · pp. 1–2
• Panoptic Lifting [49] aligns labels with an image-dependent permutation, but its Hungarian assignment cost grows cubically with identifiers and may require an extensive canonical label space.Contrastive Lift instead supervises label-invariant distances between embeddings.Source: §1 · p. 2
• Contrastive Lift represents 3D instances with low-dimensional embeddings supervised by distances between corresponding 2D label embeddings, rather than requiring labels to align across views.The embedding dimensionality is independent of the number of objects, and the learning objective avoids assignment and has complexity independent of scene object count.Source: §1 · p. 2
• Contrastive Lift combines a contrastive loss with a slow-fast learning scheme in a neural fusion field framework.The authors describe these ideas as a first combination in neural fields.Source: §1 · p. 3
• The evaluation compares the method with recent techniques, including Panoptic Lifting [49], on ScanNet, Replica, and Hypersim, and introduces Messy Rooms scenes with up to 500 objects.Messy Rooms is described as a semi-realistic dataset for demonstrating scalability to many instances.Source: §1 · p. 3

2 Related Work

• NeRF-based scene representations support semantic modeling and scene decomposition, while 2D annotations have been used to predict semantic labels or fields in 3D.Other work fuses multi-view semantic observations in 3D or distills dense features into neural fields for applications including zero-shot 3D segmentation.Source: §2 · p. 3
• For 3D panoptic segmentation, some methods use 3D instance supervision, while Panoptic Neural Fields [32] represents instances with separate MLPs and requires prior dynamic-object tracking.Contrastive Lift targets 2D instance lifting without 3D masks or object tracks; Panoptic Lifting [49] instead uses linear assignment to align multi-view annotations.Source: §2 · p. 3
• Slow-fast contrastive learning builds on a fast student and a slowly updated teacher, with the teacher updated as a moving average of the student.The paper extends this learning idea to neural fields.Source: §2 · p. 3
• Prior segmentation work clusters pixel-level embeddings or learns pixel-cluster assignments; Contrastive Lift learns embeddings and cluster centers using volumetric rendering from 2D labels.Source: §2 · p. 3

3 Proposed Method: Contrastive Lift

• A neural field maps 3D coordinates to density, view-dependent radiance, instance embeddings, and semantic embeddings, which are projected into images by differentiable ray casting.The field is fit from multiple images with camera poses and per-image 2D instance labels; labels can be noisy and inconsistent across views.Source: §3 · p. 4
• Contrastive learning pulls together embeddings of pixels with the same segment label and separates different-label pixels, applying pairs only within each image.Within-image sampling avoids directly matching identifiers that are inconsistent between views.Source: §3 · pp. 4–5
• A slowly updated embedding field, whose parameters follow an exponential moving average of the fast field, reformulates the contrastive objective to address high-variance gradients.Pixels are partitioned into non-overlapping sets for the fast and slow fields, keeping computational cost the same as the earlier objective.Source: §3 · p. 5
• The concentration loss penalizes fast embeddings’ squared distance from the corresponding centroid predicted by the slow field, reducing cluster variance but not separating clusters by itself.Contrastive losses provide the cluster-separation objective.Source: §3 · p. 5
• Semantic prediction adds one embedding dimension per class, rendered as labels and trained with cross-entropy plus a segment-consistency loss.The consistency loss encourages predicted semantic classes to agree within an image segment.Source: §3 · p. 5
• After training, rendered fast-field embeddings are clustered with HDBSCAN, and novel-view pixels receive the label of their nearest cached centroid.The embeddings are sampled from 100 random viewpoints, not necessarily training views.Source: §3 · p. 5

4 Messy Rooms Dataset

• Realistically textured Google Scanned Objects are dropped into non-overlapping positions and settled by physics simulation before rendering from inward-facing dome-shell viewpoints.Background, floor, and lighting use 360° HDRI textures projected onto a dome.Source: §4 · p. 6
• Scenes contain 25, 50, 100, or 500 objects, with eight scenes generated at 512×512 resolution and up to 1,200 viewpoints.The viewpoint count is set by the specified rule based on object count.Source: §4 · p. 6
• Floor area, camera distance, and focal length are scaled with scene size to keep the number of visible objects per image comparable.This design limits changes in per-image object distribution as the total object count grows.Source: §4 · p. 6
• Rendered instance IDs provide ground-truth maps consistent across views, while evaluation groups all object types into one foreground class.The foreground grouping focuses evaluation on instance segmentation rather than semantic classes.Source: §4 · p. 6

5.1 Results

• +3.9, +1.4, and +0.8 PQscene points over Panoptic Lifting on ScanNet, Hypersim, and Replica, respectively; Slow-Fast consistently outperforms baselines across all three datasets.Table 1 reports PQscene; prior-work performance is sourced from [49].Source: Table 1 · p. 7, §5.1 · pp. 7–8

5 Experiments

• Evaluations use ScanNet, Hypersim, Replica, and Messy Rooms; the reported scene-level metric, PQscene, accounts for instance-ID consistency across views.Tracked ground-truth instance maps are used only for evaluation, not training.Source: §5 · pp. 6–7

5.1 Results

• On Messy Rooms scenes with 25, 50, 100, and 500 objects, Contrastive Lift's improvement over PanopLi is larger, and its rendered segmentations are clearer and cross-view consistent despite noisy, inconsistent Detic segments.Figure 4 marks regions where PanopLi performs poorly and where both methods perform poorly.Source: Table 2 · p. 7, §5.1 · p. 8, Figure 4 · p. 8

5.2 Ablations

• An instance embedding size of 3 is already almost optimal on ScanNet, Hypersim, and Replica; experiments use size 24 there and size 3 on Messy Rooms.Figure 5 plots embedding size against PQscene performance.Source: §5.2 · p. 8, Figure 5 · p. 9
• The slow-fast embeddings cluster more compactly than vanilla contrastive embeddings and are easier to distinguish with post-processing such as HDBSCAN.The ablations also compare slow-fast with and without concentration loss, vanilla contrastive, and vanilla with fast-field concentration loss.Source: §5.2 · pp. 8–9, Figure 6 · p. 9
• Contrastive Lift significantly improves over the underlying 2D model even when matching baselines use ground-truth 3D information for tracking.The last two tracking baselines use 3D ground truth; one iteratively fuses segments in a point cloud.Source: §5.2 · p. 9, Table 4 · p. 10
• Contrastive Lift also improves per-frame segmentation quality, measured by Panoptic Quality, in addition to producing tracked segmentations.The evaluation trains the method using segments from Mask2Former, MaskFormer, and Detic.Source: §5.2 · p. 9, Table 5 · p. 10
• PanopLi iterations slow as its object-count cap K increases, while Contrastive Lift is agnostic to object count and its training speed is largely unaffected by embedding size.The speed comparison measures iterations per second on an NVIDIA A40 GPU while optimizing the instance field and freezing other fields.Source: §5.2 · p. 10, Table 6 · p. 10

• The method improves noisy 2D input segmentations but cannot recover from catastrophic failures such as entirely missing object classes.Source: §6 · p. 10
• Because reliable 3D reconstruction is required and remains unreliable in dynamic settings, the study focuses on static scenes.Source: §6 · p. 10
• The work reports no particular direct societal impact; its datasets are research-licensed and contain no personal data.Source: §6 · p. 10

• Contrastive Lift is more accurate and scalable than approaches that explicitly seek compatibility among multi-view labels.Source: §7 · p. 10
• The method learns a 3D vector field that characterizes the scene's object instances from 2D segmenter outputs fused using 3D neural fields.Source: §7 · p. 10
• Fitting is invariant to permutation of object labels assigned independently and arbitrarily in each input image, supporting work on larger object collections.Source: §7 · p. 10

• The representative Messy Rooms scenes span 25, 50, 100, and 500 objects, with similar object-per-image density across that range.Source: A Messy Rooms dataset · p. 15
• The dataset includes corresponding 3D scenes used to generate its examples.Source: A Messy Rooms dataset · p. 15
• The full Messy Rooms dataset is available through the link given in the paper.Source: A Messy Rooms dataset · p. 15

B Implementation Details

• The appendix provides further implementation details in addition to those in Sections 3 and 5.Source: B Implementation Details · p. 15

B.1 Architectural details

• Density and color grids start at 128×128×128 resolution and increase to 192×192×192 by the end of training, using 16 and 48 components respectively.Source: §B.1 · p. 15
• Fast and slow instance-field MLPs use hidden size 256, while the semantic-field MLP uses hidden size 128.Source: §B.1 · p. 15

B.2 Training details

• RGB reconstruction trains for 40k iterations, semantic loss starts at 40k, instance embedding loss at 160k, and segment consistency at 280k (200k for Hypersim).Loss weights are 1.0, 0.1, 0.1, and 1.0; the authors report final performance is not very sensitive to these choices.Source: §B.2 · p. 15

B.3 Post-processing Clustering details

• HDBSCAN clusters embeddings separately within predicted semantic classes, avoiding a required object-count input and using compact clusters for centroid assignment.The minimum cluster size is selected by a sweep on 10% of training data because expected samples per object decrease as object count rises.Source: §B.3 · pp. 15–16

C Comparison between different clustering algorithms.

• MeanShift and DBSCAN perform slightly worse but remain comparable to HDBSCAN on PQscene.MeanShift struggles to converge above embedding size 10; the comparison uses size 3 for fairness.Source: Table 7 · p. 16, C Comparison between different… · p. 16

D Quality of our semantic and radiance field

• On Messy Rooms, both methods use the same density and semantic model, leaving instance-field quality as the factor influencing final performance.The fields are pre-trained identically before each method trains its instance field with its respective objective.Source: D Quality of our semantic and radiance… · p. 16
• Rendered mIoU and PSNR are nearly the same as Panoptic Lifting.Table 8 reports mean Intersection over Union (mIoU) and peak-signal-to-noise ratio (PSNR); the main tables evaluate instance-map quality and consistency.Source: Table 8 · p. 16, D Quality of our semantic and radiance… · p. 16

E Comparisons to other metric learning loss functions

• Both vanilla contrastive and slow-fast objectives outperform the tested alternative metric-learning losses.The comparison uses embedding size 3 and reports PQscene; the DINO-style baseline performs worse than metric-learning losses on ScanNet and is not evaluated on Messy Rooms.Source: Table 9 · p. 17, E Comparisons to other metric learning… · p. 17

F Stability of Slow-Fast loss compared to Vanilla contrastive loss

• The slow-fast loss keeps relative gradient variance in a more controlled range than vanilla contrastive loss, which exhibits spikes.The authors attribute the reduced variance to the slowly updated embedding field.Source: F Stability of Slow-Fast loss compared… · p. 17, Figure 7 · p. 18

G More qualitative visualizations

• Instance labels are consistent across views, and the embedding heatmaps peak at object locations; the 50-object visualization shows distinctly separable instances.The heatmaps plot normalized negative log distance from selected cluster centroids; instance labels are computed only for predicted “thing” pixels.Source: G More qualitative visualizations · pp. 17–18, Figure 13 · p. 20