PaperlayerView exact revision on arXiv

Source-linked AI summary

AQ3D: Adaptive Query Transformer for 3D Instance Segmentation

Keno Moenck, Thorsten Schüppstuhl

arXiv:2608.30618v1cs.CV

TL;DR

AQ3D addresses fixed query budgets and training-bound positional encodings in 3D instance segmentation, where indoor scenes vary in size and object count. It scales queries with scene superpoints, uses 3D RoPE and decoder refinements, and reports state-of-the-art performance across three ScanNet benchmarks. Its scale-consistent design trades some expressiveness for reduced dependence on scene-size assumptions, while variable query counts create batching overhead.

  • Problem

    Indoor scans vary in spatial extent and object count, but typical decoders use fixed query counts and positional encodings tied to training-scene statistics.

  • Method

    AQ3D uses an overcomplete query set sized as a fixed ratio of scene superpoints, 3D RoPE over metric coordinates, attribution pooling, mask refinement, and cosine classification.

  • Results

    AQ3D sets a new state-of-the-art on validation and hidden test splits across ScanNetV2, ScanNet200, and ScanNet++V2 among decoder methods without additional data augmentation.

  • Takeaways & Limitations

    Scene-adaptive queries and scale-consistent positional modeling support 3D instance segmentation across scenes with varying extent and object count.

  • Takeaways & Limitations

    AQ3D trades expressiveness for scale-consistency, and variable query counts increase padded batching cost compared with fixed-budget MAFT.

Abstract

from arXiv · show

Transformer-based decoders for 3D instance segmentation typically commit to a fixed number of queries and positional modeling calibrated on the training distribution rather than on the scene at hand. Indoor scans vary widely in spatial extent and object count, so a fixed query set over-initializes small scenes and under-initializes large ones, while learned absolute and relative encodings are bound to the training scenes' extents and can saturate. We present AQ3D, which is designed to handle scenes of various sizes during training and inference. Queries are instantiated at a fixed ratio of the scene's superpoints, forming an overcomplete set whose background rejection is entirely left to the decoder. Positional information is encoded using 3D RoPE over quantized metric coordinates, replacing learned bounded lookup tables of prior decoders. Further, we improve the decoder itself by using attribution-based superpoint pooling, a mask refinement branch, and a cosine classifier for background rejection. Experiments show our method sets a new state-of-the-art on validation and hidden test splits across the datasets ScanNetV2, ScanNet200, and ScanNet++V2 among decoder methods trained without additional data augmentation. Code is available at \href{https://github.com/kenomo/aq3d}{github.com/kenomo/aq3d}.

1. Introduction

AQ3D addresses scene-scale variation in 3D instance segmentation by adapting query count and positional modeling to each input scene. It uses an overcomplete, superpoint-relative query set and 3D RoPE to avoid fixed scene-agnostic designs.

  • Motivation: Indoor scans vary in spatial extent, superpoint count, and foreground instance count within and across datasets.ScanNet++V2 contains dense, high-resolution scans with greater variation and more superpoints.
  • Query initialization: Most decoders use a fixed query count for every training and inference sample, regardless of scene content.Scene-derived query methods generally collapse their initial pools back to a fixed number.
  • Positional modeling: Learned absolute and relative positional encodings are tied to normalized scene bounds or training-distribution statistics and can saturate outside those ranges.MAFT uses normalized coordinates and bounded lookup tables indexed by quantized relative offsets.
  • Contribution: AQ3D instantiates an overcomplete query set with NQ = floor(r · NS) and leaves background rejection to decoder layers without proposal supervision.The same ratio is applied during training and inference, unlike OneFormer3D’s training-only subsampling.
  • Contribution: AQ3D replaces prior absolute and learned contextual relative encodings with 3D RoPE to provide scale-consistent spatial reasoning.The method applies RoPE over quantized metric coordinates and reports state-of-the-art results across ScanNetV2, ScanNet200, and ScanNet++V2 among comparably trained decoder methods.

2. Related Works

3D instance segmentation has shifted toward transformer decoders that represent objects with instance queries, while query initialization and positional encoding remain active design choices. Prior methods vary from learned queries to scene-derived or semantically conditioned queries and use several positional strategies.

  • Transformer decoders: Transformer-based 3D instance segmentation follows DETR set prediction and Mask2Former mask classification, replacing hand-tuned voting or clustering.Instance queries are refined through attention and decoded into masks and class labels.
  • Transformer decoders: Subsequent work improves backbones, augmentation, feature discrimination, and relational modeling in cross- or self-attention.These improvements target different parts of the transformer-decoder pipeline.
  • Query initialization: Query initialization ranges from fully parametric queries to scene-derived positions, explicit anchors, and semantic conditioning.SPFormer uses learned queries, Mask3D samples positions by FPS, and other methods condition queries on scene semantics.
  • Query initialization: Except for OneFormer3D, prior methods generally fix query count independently of the input scene; OneFormer3D ties it to superpoints only at inference.Its ratio acts as training-time augmentation, so the decoder processes the full superpoint set at test time.
  • Positional modeling: Positional information is injected through absolute encodings, learned relative biases, or RoPE-based rotations of query and key projections.3D methods include learned conditional encodings, PointROPE, and 3D RoPE, while SPFormer adds no extra positional encoding.

3. Method

AQ3D extracts point and superpoint representations, initializes scene-adaptive queries, and iteratively decodes masks and classes. Its decoder combines attribution-based pooling, metric-coordinate 3D RoPE, refinement, cosine classification, and supervised mask and position losses.

  • Architecture: The backbone maps colored, normal-equipped point clouds to point features, which are pooled into superpoint and mask features for parallel decoding.Precomputed superpoints partition point indices, and decoder layers refine queries and mask features.
  • Prediction heads: The class head uses learned cosine-classifier prototypes, while mask scores come from query–mask-feature dot products and a score head predicts mask IoU.Final predictions select the top-k foreground class-mask combinations after lifting masks to point resolution.
  • Backbone and Pooling: Attribution pooling assigns each point a feature-dependent scalar score and forms superpoint features from attribution-weighted constituent point features.The simplified pooling avoids pairwise interactions within a superpoint.
  • Adaptive Queries: AQ3D initializes NQ = floor(r · NS) queries by FPS on superpoint centroids, with content projected from pooled local superpoint features.Batches are padded to their maximum query count, and padded queries are masked and excluded from matching and top-k selection.
  • Decoder: Each decoder layer applies self-attention, query-to-superpoint cross-attention, and an FFN, with auxiliary center regression periodically updating query positions.Mask features are refined while superpoint features are not, and contrastive supervision is omitted without degrading performance in the reported experiment.
  • Positional Modeling: 3D RoPE rotates query and key projections along the three spatial axes, injecting relative positional offsets without learned bounded lookup tables.AQ3D relies on relative positional modeling within attention operations.
  • Loss: The objective applies deep supervision after every decoder layer, combining classification, mask BCE and Dice, IoU-score, and centroid-distance losses.Unmatched queries receive the non-object label, while other terms apply only to matched pairs and padded queries are excluded.

4. Experiments

AQ3D is evaluated across three indoor benchmarks, query-budget and scene-scale settings, and component ablations. It achieves strong validation and hidden-test results while adaptive queries and 3D RoPE address scene variation.

  • Evaluation setup: AQ3D is evaluated on ScanNetV2, ScanNet200, and ScanNet++V2 without test-time or context-based training augmentation.The implementation uses a sparse U-Net or Volt backbone, six decoder layers, and dataset-specific positional-modeling choices.
  • Benchmark results: AQ3D ranks first on both listed hidden-test metrics for ScanNetV2 and uses half the parameters of SPFormer + Volt-B.The 44M-backbone model also improves the best previously reported validation result by +2.2 mAP.
  • Benchmark results: AQ3D improves over sparse-backbone competitors on ScanNet200 and achieves the best reported validation results when using Volt-B.With the 44M sparse backbone, it matches SPFormer + Volt-B with ACGP on mAP while using half the parameters and no context-based or test-time augmentation.
  • Benchmark results: +2.0 mAP and +0.8 AP@50 separate AQ3D from SPFormer + Volt-B on the ScanNet++V2 hidden test split.AQ3D remains within 0.8 mAP of the ACGP-augmented variant, which uses context-based training augmentation.
  • Query budget: On ScanNetV2, performance saturates for query ratios r > 0.6, while mean Recall continues rising as denser initialization discovers more instances.Below r = 0.4, mRC and mAP decline together, indicating unrecovered instances; higher coverage makes query discrimination harder.
  • Scene scale: On full-scale ScanNet++V2 scenes, MAFT loses ≈5 points on mAP and AP@50, whereas AQ3D loses only ≈1.Training scenes average 7.1±1.1 m, while unchunked validation scenes average 8.1 ± 3.3 m in xy-axis-aligned diagonal.
  • Query rejection: A ScanNetV2 scene initializes 418 queries inside the top-15 prediction masks, but only 23 survive non-object labeling, scoring, and NMS.The visualization shows the 15 highest-scoring predictions localized on or inside their predicted objects, with most nearby queries rejected.
  • Ablations: +1.9 AP@50 is the largest single gain from replacing MAFT relative encodings with 3D RoPE, while adaptive queries provide the largest single mAP gain of +1.3.RoPE is especially effective on ScanNet++V2; its contribution also grows with instance density, and removing positional signals costs −3.1 mAP and −3.8 AP@50 on ScanNet200.

5. Conclusion

AQ3D adapts query count and positional modeling to scene size, using superpoint-relative queries and metric-coordinate 3D RoPE. This improves scale consistency but introduces compute, calibration, and recall constraints.

  • 5. Conclusion: AQ3D instantiates queries at a fixed ratio of scene superpoints without proposal supervision, replacing fixed query sets with scene-adaptive initialization.The overcomplete query set leaves background rejection to decoder layers.
  • 5. Conclusion: 3D RoPE over quantized metric coordinates keeps spatial reasoning scale-consistent while removing bounded positional lookup tables.The design exchanges some learned expressiveness for scale consistency.
  • 5. Conclusion: On full-scale ScanNet++V2 scenes, MAFT loses roughly 5 mAP while AQ3D loses about 1 mAP.The comparison concerns performance degradation on full-scale scenes.
  • 5. Conclusion: AQ3D’s highest costs are quadratic self-attention growth with superpoint count and dependence on dataset-specific recalibration of the superpoint segmentator.The segmentator is unlearned and can upper-bound recall through instances lost to superpoint partitioning.

Supplementary Material

Table 9 reports independent class-head and mask-refinement ablations on ScanNetV2 and ScanNet200 validation sets.

  • Supplementary Material: Table 9 evaluates independent single-factor changes to class-head and mask-refinement baselines on ScanNetV2 and ScanNet200 validation sets.The table caption defines the comparison structure.

6. Additional Ablations

Additional ablations show that cosine background classification, mask refinement, non-object weighting with dropout, and fine RoPE quantization materially affect performance. Their effects vary across datasets and settings.

  • Class Head: Replacing the cosine classifier with a linear projection costs 0.7 mAP on ScanNetV2 and 1.9 mAP and 2.5 AP@50 on ScanNet200.Cosine normalization keeps background rejection dependent on direction rather than feature magnitude.
  • Mask Refinement: Removing mask refinement costs 0.9 mAP and 0.7 AP@50 on ScanNetV2, versus 4.1 mAP and 3.4 AP@50 on ScanNet200.The larger ScanNet200 drop is attributed to mask tokens helping object discrimination.
  • Training Recipe: Increasing λ∅ and adding dropout together adds 1.4 mAP, because higher non-object weight strengthens background-prediction training signal while dropout counteracts overfitting.The two factors are not separable: λ∅=0.5 needs more dropout, whereas λ∅=0.1 is preferable without dropout.
  • Training Recipe: Table 10 measures ScanNetV2 validation differences relative to baseline IV while varying λ∅ and dropout.λ∅ denotes the non-object weight.
  • RoPE Hyperparameters: Table 11 measures ScanNetV2 validation AP differences relative to the θ = 100, 0.05 baseline while varying RoPE frequency and grid size.The caption identifies θ and grid size as the varied hyperparameters.
  • RoPE Hyperparameters: Coarsening RoPE quantization from 0.05 m to 0.1 m degrades performance at every θ, while the base frequency has comparatively small effects.The retained setting is θ=100 with a 0.05 m grid.

7. Compute

AQ3D’s adaptive query count changes decoder compute substantially: self-attention becomes more expensive, while RoPE can reduce cross-attention cost at equal budgets. Variable query counts also create batching overhead.

  • 7. Compute: The compute profile accumulates forward-pass MACs for self-attention, cross-attention, and FFN over one training epoch’s augmented scenes.Only decoder modules whose cost scales with query count are compared.
  • Decoder Cost: At batch size 1, AQ3D costs 6.8 GMACs per ScanNetV2 scene versus MAFT’s 4.6, but 10.3 versus 11.5 on ScanNet++V2.AQ3D is therefore more expensive on ScanNetV2 and slightly cheaper on ScanNet++V2 in this comparison.
  • Decoder Cost: AQ3D’s adaptive scheme makes self-attention more expensive, while its RoPE-only ScanNet++V2 configuration reduces cross-attention width and GMACs.The FFN is roughly identical between methods at equal query budgets because it scales linearly with query count.
  • Batching Overhead: Batch size four raises AQ3D cost by 56% and 50%, compared with 19% and 16% for MAFT, because variable query counts require padded masked entries.Grouping scenes with similar superpoint counts can reduce padding but correlates batch composition with scene size.

8. Scene Statistics

Scene complexity and instance count rise with superpoint count across datasets, while ScanNet++V2 spans the widest scene-size range and exposes fixed query budgets most strongly.

  • Scene Statistics: AQ3D uses the number of superpoints, NS, as a proxy for geometric complexity and spatial extent.The design premise is that foreground-object counts grow with scene complexity and extent.
  • Scene Statistics: Instance counts increase with NS across all three datasets and both training and validation splits.ScanNetV2, ScanNet200, and ScanNet++V2 all show a positive relationship between instance count and superpoint count.
  • Scene Statistics: ScanNet++V2 validation scenes exceed 10 000 superpoints and 200 instances, with substantially greater scene-size variance.This is the setting where a fixed query budget is most severely miscalibrated.
  • Scene Statistics: ScanNet200 has roughly twice as many instances as ScanNetV2 despite using the same reconstructions and unchanged NS.Its finer 200-class taxonomy produces more instances per scene.

9. ScanNet++V2 Segmentator Configuration

ScanNet++V2 superpoints are recalibrated to control decoder cost while limiting instance loss, selecting nv = 100 and a merging cutoff near kt ∈[0.1, 0.2].

  • Configuration rationale: ScanNet++V2’s sub-millimeter reconstruction creates excessive superpoints, increasing both cross-attention cost and the adaptive query budget.Because NQ = ⌊r · NS⌋, inflated NS also raises self-attention cost.
  • Configuration objectives: The configuration minimizes both average superpoint count and instances lost through majority-vote labeling during training.Coarser superpoints reduce NS but can merge small instances into neighboring objects.
  • Minimum segment size: 8.9 instances are lost on average at nv = 300, compared with approximately 1.6 at nv = 50, so nv = 100 is selected.Instance loss grows monotonically as minimum segment size increases.
  • Merging cutoff: 3.15 and 3.16 instances are lost on average at kt ∈[0.1, 0.2], while average NS decreases from 2 676 at kt = 0.02 to 2 358 at kt = 0.5.The loss minimum forms a plateau near kt ∈[0.1, 0.2], whereas coarser merging steadily reduces superpoint count.

10. Qualitative Results on ScanNet++V2

On full-scale ScanNet++V2 scenes, AQ3D qualitatively retrieves finer, more complete, and better-separated instances than the fixed-query MAFT baseline, including in small scenes.

  • Evaluation setting: The qualitative comparisons use unchunked full-scale validation scenes, which are considerably larger than the training crops.The observations are reported as consistent with the quantitative gap in Tab. 6.
  • Fine-grained objects in dense regions: AQ3D recovers small objects in dense regions that MAFT often absorbs into large supporting-furniture masks.Its adaptive budget places more queries in geometrically detailed regions with elevated NS.
  • Foreground object retrieval: At equal retained-prediction counts, AQ3D spends fewer top-ranked masks on structural surfaces such as windows, walls, and ceiling parts.Some of MAFT’s top-100 predictions cover non-instance structural classes instead.
  • Separation of neighboring instances: AQ3D separates adjacent same-class objects more reliably and produces more complete masks than MAFT.In the stool cluster, AQ3D assigns distinct instances where MAFT produces one merged mask.
  • Small scenes: On a small scene, AQ3D retrieves more of the few present objects even when it instantiates fewer queries than MAFT’s fixed budget.The instantiated queries remain grounded in the scene rather than competing with an oversized fixed slot pool.
Loading 2608.30618v1…