Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

I. INTRODUCTION

• Prior 3D methods rely on manually chosen voting targets and clustering heuristics, while Transformer-based 3D instance segmentation had not yet been explored.These methods typically form masks by grouping votes and use proxy losses rather than directly predicting and supervising instance masks.Source: §I · pp. 1–2
• Mask3D uses sparse-convolution point features and coarse-to-fine Transformer decoders to produce semantic instance queries whose features yield masks by scoring all points.The queries encode semantic and geometric information; similarity scores form point-cloud heatmaps that are normalized and thresholded into binary masks.Source: §I · p. 2
• Before applying mask loss, Mask3D uses bipartite graph matching to optimally associate predicted and annotated masks.Unlike nearest-ground-truth matching, this avoids non-optimal associations and ensures unmatched annotated masks are not omitted from the loss.Source: §I · p. 2
• Mask3D is the first competitive Transformer-based model for 3D semantic instance segmentation and achieves state-of-the-art performance on ScanNet, ScanNet200, S3DIS, and STPLS3D.The introduction identifies these as the four evaluated datasets.Source: §I · p. 2

3D Instance Segmentation

• Prior work spans bottom-up [28] [32] [58] [59], top-down [22] [61], and voting-based [4] [26] [56] methods.Source: §II · p. 2
• MASC [32] clusters pairwise affinities offline; DyCo3D [21] predicts masks with dynamic convolutions but lacks optimal supervision assignment, while 3D-BoNet [61] predicts instances in parallel from one scale.MASC uses a multi-scale hierarchical feature backbone, whereas 3D-BoNet's single-scale scene descriptor cannot encode masks of diverse sizes.Source: §II · p. 2

Transformers

• Earlier 3D Transformer work largely addressed detection [34] [37] [39] or semantic segmentation [29] [40], often modifying attention or using Transformers for proposal refinement or backbone context.Misra et al. [37] applied a vanilla Transformer to point clouds but relied on learned downsampling; DyCo3D [21] used a Transformer at the feature-backbone bottleneck rather than for instance-mask prediction.Source: §II · p. 2
• Mask3D applies a vanilla Transformer decoder to 3D semantic instance segmentation and achieves state-of-the-art performance.Source: §II · p. 2

Mask Module

• The mask module thresholds sigmoid dot-product scores at 0.5 for binary masks and predicts each query’s class, or inactivity, with a C + 1-way softmax.Masks are generated at each decoder layer and pooled to the voxel-feature resolution for the next masked cross-attention step.Source: §III · p. 3, Figure 2 · p. 3

Sparse Feature Backbone

• A sparse convolutional U-net backbone quantizes colored points into voxels and projects multiresolution features to a shared dimension for query attention and full-resolution mask prediction.Queries attend to coarser feature maps, while the full-resolution map supports auxiliary and final instance masks.Source: §III · p. 3

Query Refinement

• Queries refine through 12 steps across the four coarsest feature levels, using masked cross-attention and inter-query self-attention to limit duplicate masks.The decoder cycles through feature levels from coarse to fine three times, sharing layer weights across iterations.Source: §III · pp. 2–4

Sampled Cross-Attention

• Sampled cross-attention pads and masks variable voxel counts, resamples voxels at each decoder layer when counts exceed a threshold, and uses all voxels at inference.This saves memory and enables higher-resolution grids, including 2 cm voxel side-length on ScanNet [9].Source: §III · p. 4

Correspondences

• Hungarian matching assigns predicted instances to ground truth by minimizing a weighted cost of mask Dice, mask BCE, and class cross-entropy losses.The weights are λdice = 2.0, λBCE = 5.0, and λcl = 2.0.Source: A. Training and Implementation Details · p. 4
• Assigned masks use binary cross-entropy and Dice losses, while class cross-entropy supervises labels and weights unassigned no-object predictions by 0.1.Auxiliary instance predictions after each decoder layer are also included in the overall loss.Source: A. Training and Implementation Details · p. 4

Prediction Confidence Score

• Mask3D scores instances using query and point features, combining softmax class confidence with mean confidence over voxels inside the binarized mask, rather than a dedicated ScoreNet [26].The mask-confidence component follows Mask2Former [5].Source: A. Training and Implementation Details · p. 4

Query Types

• Non-parametric queries initialized from sampled point locations improve performance and allow query counts to change at inference, trading speed for performance without retraining.The paper also experiments with using sampled point features as query features; the described initialization uses zero query features and sampled-point positional encodings.Source: A. Training and Implementation Details · p. 4

Training Details

• Mask3D uses a Minkowski Res16UNet34C backbone and trains for 600 epochs with AdamW and a one-cycle learning-rate schedule.The maximum learning rate is 10^-4; 1000 epochs did not further improve results. Training with 2 cm voxelization takes about 78 hours on an NVIDIA A40 GPU. For ScanNet, query-feature dot products are computed within graph-based segments to reduce memory use; augmentations include flipping, z-axis rotations, elastic distortion, scaling, and color jitter, brightness, and contrast changes.Source: A. Training and Implementation Details · p. 4

IV. EXPERIMENTS

• The evaluation compares Mask3D with prior state-of-the-art approaches on four publicly available 3D indoor and outdoor datasets.Source: §IV · p. 4
• On ScanNet v2, Table I reports mAP at different IoU thresholds across 18 classes for validation and test sets, plus validation inference speed on a TITAN X GPU excluding postprocessing.Source: Table I · p. 5
• The analysis examines query types, query-refinement steps, and inference query counts, followed by qualitative results and limitations.Source: §IV · p. 5

A. Comparing with State-of-the-Art Methods

• Evaluation uses mAP across the benchmarks; S3DIS also reports mPrec50/mRec50 and filters predictions below 80% confidence to limit false positives.S3DIS results cover Area 5 and six-fold cross-validation; ScanNet200 tests long-tail class imbalance, and STPLS3D is an outdoor dataset.Source: A. Comparing with State-of-the-Art… · p. 5
• Mask3D exceeds prior work by at least 6.2 mAP on ScanNet, 6.2 mAP on S3DIS, 10.8 mAP on ScanNet200, and 11.2 mAP on STPLS3D.The reported S3DIS and ScanNet200/STPLS3D comparisons are presented in Tables II and III, respectively.Source: Table II · p. 5, Table III · p. 5, A. Comparing with State-of-the-Art… · p. 5
• ScanNet pre-training improves Mask3D by 1.2 mAP on S3DIS Area 5, while strong results across indoor, outdoor, and imbalanced-class settings require no architecture or training-regime modifications.The evaluation spans ScanNet, ScanNet200, S3DIS, and STPLS3D.Source: A. Comparing with State-of-the-Art… · p. 5

Mask Loss

• On ScanNet validation, a weighted sum of Dice and binary cross-entropy gives the best mask-loss performance, improving over binary cross-entropy alone.Dice loss is designed to address the imbalance between foreground and background mask points.Source: Table IV · p. 6, B. Analysis Experiments · p. 6

Number of Queries and Decoders

• Training with K = 200 rather than K = 100 slightly improves performance at extra memory; fewer inference queries hurt but run faster, while more typically add less than 1 mAP.The final model uses K = 100 because of memory constraints with 2 cm voxels in the feature backbone.Source: Figure 3 · p. 6, B. Analysis Experiments · p. 6
• Mask quality rises rapidly through 4 Transformer decoder layers, then improves more slowly.The analysis measures mask quality after each decoder layer in the trained model.Source: Figure 3 · p. 6, B. Analysis Experiments · p. 6

Query Types

• On ScanNet validation, FPS position-only non-parametric queries outperform learned parametric queries, while adding sampled-point features degrades performance versus both.The position-only setup initializes queries to zero; the feature-augmented setup uses features at the sampled positions.Source: B. Analysis Experiments · p. 6, Table IV · p. 6

C. Qualitative Results and Limitations

• Mask3D shows robust instance segmentation across diverse ScanNet scenes with clutter, scanning artifacts, and many similar objects.The representative examples are shown in Fig. 4.Source: C. Qualitative Results and Limitations · p. 6
• A systematic limitation is that Mask3D can merge instances that are far apart in the scene.One example appears in Fig. 4, bottom left.Source: C. Qualitative Results and Limitations · p. 6
• Full-point-cloud attention can combine distant objects with similar semantics and geometry when they have similar learned point features, a failure less likely with explicit geometric priors.Source: C. Qualitative Results and Limitations · p. 6

V. CONCLUSION

• Mask3D’s Transformer decoders learn instance queries that combine with point features to directly predict semantic instance masks, without hand-selected voting schemes or hand-crafted grouping mechanisms.The authors describe this as an alternative to approaches that rely on voting and geometric grouping.Source: §V · p. 6
• The authors expect follow-up work on Transformer-based approaches to 3D semantic instance segmentation.They characterize Mask3D as an attractive alternative to current voting-based approaches.Source: §V · p. 6

• The full Mask3D model uses five-scale point features and iterative Transformer query refinement; its mask module predicts semantic classes and thresholded instance heatmaps.The main paper’s illustration showed fewer hierarchical backbone levels and query-refinement layers than the full model.Source: Figure 5 · p. 9
• The supplementary material reports experiments with convolutional and Transformer-based feature backbones.Source: Table V · p. 9

I. IMPLEMENTATION DETAILS

• For S3DIS, training uses randomly cropped 6m×6m blocks for 1000 epochs, while testing disables cropping and processes full scenes.The cropping addresses scenes with several million high-density points and keeps memory requirements in bounds.Source: §I · p. 9
• For STPLS3D, Mask3D processes 54m×54m blocks but retains predictions relevant to the evaluated 50m×50m block, improving results by roughly 1.2 mAP.The larger input blocks help when instances are separated across blocks by the evaluation protocol.Source: §I · p. 9
• The model uses a Minkowski Res16UNet34C backbone with five feature scales, projected to 128 dimensions, and a modified Mask2Former decoder with 100 instance queries.It attends to backbone features three times with shared decoder weights, uses eight-headed attention and a 1024-dimensional feedforward network, and omits dropout.Source: §I · p. 9
• More than 90% of Mask3D’s parameters come from the feature backbone, while the instance-segmentation-specific portion is less than 10%.The Transformer decoder is larger in absolute parameter count than the instance-specific parts of the baselines, but remains small relative to the feature backbones.Source: §I · pp. 9–10
• 40.9 vs. 40.0 mAP on ScanNet validation 5 cm shows Res16UNet18B performs comparably to Res16UNet34C, while StratifiedFormer runs but trails the sparse-convolution backbone.The smaller-backbone experiment tests whether Mask3D’s improved performance comes from having more parameters; the paper also reports that Mask3D runs with different backbone types.Source: §I · pp. 9–10

A. Comparison to SoftGroup

• DBSCAN postprocessing splits each Mask3D instance mask into spatially contiguous clusters treated as new masks, with best results at ε=0.9 for ScanNet, 0.6 for S3DIS, and 14.0 for STPLS3D.Source: A. Comparison to SoftGroup · p. 10
• Mask3D correctly segments a large U-shaped table and most adjacent chairs that SoftGroup misses or merges after center-voting and grouping failures.The qualitative comparison attributes SoftGroup’s errors to difficulty predicting centers for large non-convex shapes and for multiple nearby chairs.Source: Figure 6 · p. 11