Source-linked AI summary
Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, Katerina Fragkiadaki
TL;DR
Existing grounding systems are limited by detector proposals that may miss the entities named by an utterance. BUTD-DETR combines language and bottom-up proposal guidance while decoding objects directly, and it achieves state-of-the-art 3D results with competitive 2D performance.
Problem
Proposal-bottlenecked grounding cannot recover objects or parts that a bottom-up detector misses, including small, occluded, or rare entities.
Method
BUTD-DETR attends to language, visual features, and detector-derived box proposals, decodes objects with a detection head, and augments supervision with detection prompts.
Results
BUTD-DETR sets new state-of-the-art results on three 3D benchmarks, with gains of 12.6% on SR3D, 11.6% on NR3D and 6.3% on ScanRefer, while remaining competitive with MDETR in 2D.
Takeaways & Limitations
The model demonstrates that language-modulated detection with bottom-up proposal cues can support referential grounding in both 3D point clouds and 2D images.
Takeaways & Limitations
The method does not robustly ground abstractions such as viewpoint-dependent left and right, precise counting, or logical constraints on out-of-distribution visual input.
Abstract
from arXiv · showhide
Most models tasked to ground referential utterances in 2D and 3D scenes learn to select the referred object from a pool of object proposals provided by a pre-trained detector. This is limiting because an utterance may refer to visual entities at various levels of granularity, such as the chair, the leg of the chair, or the tip of the front leg of the chair, which may be missed by the detector. We propose a language grounding model that attends on the referential utterance and on the object proposal pool computed from a pre-trained detector to decode referenced objects with a detection head, without selecting them from the pool. In this way, it is helped by powerful pre-trained object detectors without being restricted by their misses. We call our model Bottom Up Top Down DEtection TRansformers (BUTD-DETR) because it uses both language guidance (top down) and objectness guidance (bottom-up) to ground referential utterances in images and point clouds. Moreover, BUTD-DETR casts object detection as referential grounding and uses object labels as language prompts to be grounded in the visual scene, augmenting supervision for the referential grounding task in this way. The proposed model sets a new state-of-the-art across popular 3D language grounding benchmarks with significant performance gains over previous 3D approaches (12.6% on SR3D, 11.6% on NR3D and 6.3% on ScanRefer). When applied in 2D images, it performs on par with the previous state of the art. We ablate the design choices of our model and quantify their contribution to performance. Our code and checkpoints can be found at the project website https://butd-detr.github.io.
1 Introduction
BUTD-DETR addresses the proposal bottleneck in language grounding by combining language-directed and objectness-driven attention while decoding referenced objects rather than selecting from detector proposals. It achieves strong 3D benchmark gains and competitive 2D performance.
- Motivation: Language-directed perception adapts visual processing to the utterance, helping localize objects that task-agnostic bottom-up perception may miss.The utterance can direct attention to a visually nonsalient bottle.
- Motivation: Existing grounding models select referents from detector proposals, so they cannot recover small, occluded, rare, or otherwise missed objects.Examples include a missed clock on a shelf and bottle on a bathroom vanity.
- Method: BUTD-DETR attends to language and detector-derived box proposals but decodes objects with a detection head instead of selecting from the proposal stream.The model uses bottom-up box proposals as an additional input stream while avoiding a box bottleneck.
- Method: BUTD-DETR augments grounding supervision by treating object-category label lists as detection prompts and grounding labels that correspond to objects in the scene.This includes learning to discard labels mentioned in prompts that have no corresponding scene objects.
- Results: 12.6% on SR3D, 11.6% on NR3D and 6.3% on ScanRefer are the reported 3D gains over prior approaches, while 2D performance is competitive with MDETR.The model also requires less than half of MDETR’s GPU training time in 2D because of cheaper deformable attention.
- Results: The paper reports state-of-the-art performance across 2D and 3D scenes and uses ablations to quantify the contributions of its design choices.The stated contributions include detection prompts, an additional box stream, and improved box–word-span alignment losses.
2 Related work
Prior 2D and 3D grounding systems commonly rely on object proposals and score candidate regions, whereas BUTD-DETR removes this proposal bottleneck and evaluates 3D grounding without oracle boxes.
- Object detection with transformers: DETR-family object detectors decode boxes and labels with object queries, while deformable attention reduces the quadratic cost of pixel-to-pixel self-attention.These detection-transformer ideas provide the architectural basis for visual feature extraction.
- 2D referential language grounding: Most 2D grounding methods extract object proposals and fuse them with language, although some directly regress or deform a target box without proposal selection.The related approaches differ in how language and visual features are fused and how boxes are predicted.
- 3D referential language grounding: Related 3D grounding approaches represent proposals, encode language, and fuse modalities before scoring each proposal as the answer.Their pipelines use point features, segmentation masks, spatial or categorical features, graph networks, or Transformers.
- 3D referential language grounding: 3D benchmarks often provide ground-truth object boxes at test time because 3D detectors perform worse than their 2D counterparts.This benchmark convention reflects the difficulty of 3D object detection.
- 3D referential language grounding: BUTD-DETR is the first reported 3D grounding model evaluated on Referit3D without access to oracle 3D object boxes.It therefore evaluates grounding with detected objects rather than benchmark-provided oracle boxes.
3 Method
BUTD-DETR grounds referential utterances in 2D images and 3D point clouds by jointly attending to visual, language, and detector-provided box streams, then decoding object boxes and aligned language spans. It augments grounding supervision with synthetic detection prompts and uses detector-aware training strategies to address train–test prediction discrepancies.
- Training objectives: Training combines box regression, token prediction, and bidirectional contrastive alignment between object queries and referring language spans.Predicted boxes are assigned to ground truth using Hungarian matching before these objectives are computed.
- Input encoding: The model uses pre-trained visual, language, and object detectors to produce image or point-cloud, word, and box-proposal features.RGB images use ResNet101, point clouds use PointNet++, utterances use RoBERTa, and detector proposals are encoded with semantic category labels.
- Robustness: BUTD-DETR mitigates reliance on noisier test-time 3D detector predictions by randomly replacing 30% of detected boxes during training.The detector is more accurate on the ScanNet training set than at test time, creating a train–test discrepancy.
- Cross-modality encoder: BUTD-DETR encodes visual, language, and labelled box-proposal tokens in separate streams that interact through cross-attention.In 2D, visual-stream attention can be deformable; in 3D, vanilla attention is used because point clouds are sparse and discontinuous.
- Decoder: Non-parametric visual-token queries decode object boxes and corresponding language spans instead of selecting answers from the proposal stream.Top-scoring visual tokens generate object queries, which attend to language, box proposals, and visual tokens before predicting box geometry and token spans.
- Supervision augmentation: Detection prompts cast object detection as grounding category-label utterances, including negative labels that should not match any predicted boxes.The model grounds sampled labels that appear in the scene and learns not to associate boxes with labels for absent categories.
4 Experiments
The experiments evaluate BUTD-DETR on 3D point-cloud and 2D image grounding benchmarks, compare it with existing and MDETR-3D methods, and ablate its design choices. Results show strong 3D performance and competitive 2D performance, while detector prompts and separate modality streams contribute substantially.
- Evaluation setup: The experiments cover SR3D, NR3D, and ScanRefer for 3D point clouds, plus RefCOCO, RefCOCO+, and Flickr30k for 2D images.3D experiments use separately trained models and augment supervision with ScanNet detection prompts.
- 3D comparisons: BUTD-DETR decodes boxes rather than selecting proposals and is compared with box-bottlenecked methods, MDETR-3D, and detected- or ground-truth-box evaluation setups.The 3D comparisons report top-1 accuracy at IoU@0.25, with additional IoU@0.5 results on ScanRefer.
- 3D results: BUTD-DETR outperforms existing approaches and MDETR-3D by a large margin under both detected-box and ground-truth-box evaluation.It also outperforms SAT-2D without using additional 2D image features during training.
- Ablations: 10.2% higher performance than an object-bottlenecked variant, while MDETR-3D trails BUTD-DETR by 6.7%.The object-bottlenecked variant does not attend to 3D point features or decode boxes.
- Ablations: 4.2% performance gain from co-training with detection prompts, whereas removing box-stream attention causes a 1.1% absolute accuracy drop.Replacing the proposed contrastive loss with MDETR’s causes a 2.5% absolute-accuracy drop; separate modality streams add 0.8%.
- 2D results: In 2D grounding, BUTD-DETR achieves performance comparable to MDETR while training two times faster on the reported RefCOCO and RefCOCO+ setup.The experiments use top-1 accuracy for RefCOCO and RefCOCO+, and Recall@k for Flickr30k.
5 Conclusion
BUTD-DETR grounds referential utterances in 2D and 3D scenes by jointly attending to language, visual features, and detector box proposals while decoding objects directly. It achieves state-of-the-art results in two 3D benchmarks and operates without oracle object boxes.
- Conclusion: BUTD-DETR attends to language, visual, and box-proposal streams to decode referenced objects and align them with utterance spans.The model builds on MDETR and combines bottom-up detector proposals with top-down language guidance without making proposals a bottleneck.
- Conclusion: The model outperforms its straightforward MDETR-3D equivalent by a significant margin and sets a new state of the art in two 3D language-grounding benchmarks.The reported gains are attributed to attention on labelled bottom-up proposals, detection-prompt co-training, and improved contrastive losses.
- Conclusion: BUTD-DETR is the first 3D referential-grounding model described here to detect boxes from the input point cloud instead of using oracle object boxes.This evaluates grounding in a setup without access to ground-truth object boxes.
6 Supplementary file
The supplementary material provides implementation details, additional performance analyses, and experiments on detector choices and proposal corruption. It documents the model’s 3D and 2D encoders, attention mechanisms, training settings, and evaluation tables.
- Supplementary analyses: Additional tables report performance analyses for SR3D, NR3D, and ScanRefer under their stated ground-truth or detected-box evaluation setups.The supplementary material also includes implementation and performance-analysis sections for the 3D and 2D domains.
- Implementation details: Implementation details specify PointNet++ for 3D point clouds, ResNet-101 with multi-scale features for 2D images, and spatial-categorical encodings for detected boxes.The 3D model uses 1024 visual tokens, while the 2D model uses 17,821 visual tokens of dimension 256.
- Implementation details: The models use deformable attention for visual streams and standard attention for language and detected-box streams, with different attention choices across 2D and 3D.Deformable attention is omitted in 3D because it requires pixel-feature pooling and interpolation.
- Training: Training details include frozen text encoding and separate learning rates for visual and other layers in the 3D model, plus multi-GPU pretraining for the 2D model.The 3D configuration fits batch size 6 on a 12GB GPU and batch size 24 on an NVIDIA A100.
6.3 Detailed results on SR3D/NR3D and ScanRefer
BUTD-DETR is evaluated across multiple 3D grounding protocols and scene contexts, outperforming prior methods broadly across SR3D, NR3D, and ScanRefer.
- Evaluation protocols: Evaluation uses overall accuracy, with IoU thresholds for detected-box setups and ground-truth matching for GT setups.SR3D and NR3D use GT proposals, while ScanRefer uses detected objects evaluated at 0.25 and 0.5 IoU thresholds.
- SR3D: SR3D results vastly outperform all competitors under every evaluated setup.
- NR3D: NR3D results show clear gains on all protocols except the view-dependent setup.The authors identify view prediction as a possible way to improve this setup but avoid additional domain-specific design choices.
- ScanRefer: ScanRefer results clearly outperform prior approaches under every setup except Unique@0.5, where performance matches the best competitor.
6.4 Effect of detection backbone
Changing the detection backbone affects performance, but the proposed model remains stronger than competitors and box-bottlenecked baselines.
- Detection backbone: Using VoteNet boxes on ScanRefer yields 50.0% Acc@0.25 and 37.5% Acc@0.5, versus 50.9% and 38.8% with Group-Free boxes.The VoteNet-based model still outperforms all competitors.
- Detection backbone: 46.4% on SR3D from unfreezing the detector backbone improves the baseline by 4.5% but remains 4.7% below the proposed model.The result indicates that fine-tuning does not eliminate the disadvantage of box-bottlenecked baselines.
6.5 Effect of detection augmentation
Randomly replacing detected boxes during training mitigates train-test proposal-quality differences and improves performance.
- Detection augmentation: Replacing 30% of detected boxes with random boxes during training provides a 1% absolute boost.This augmentation is applicable only when the box stream is used.
6.6 Negative training with detection prompts
The paper treats object detection as grounding category-label prompts, using positive and negative labels to augment referential-grounding supervision.
- Detection prompts: Detection prompts are category-label utterances whose mentioned object instances must be localized and aligned to their corresponding labels.
- Detection prompts: Prompts sample positive labels present in a scene together with negative labels absent from it, avoiding concatenation of an impractically large vocabulary.The method assumes object annotations when sampling scene-positive labels.
- Detection prompts: Unlike MDETR’s single-category post-pretraining prompts, this method uses longer multi-category prompts containing both positive and negative classes.
- Detection prompts: The positive-to-negative ratio is fixed at r = 1, with at most 10 positive classes sampled; tuning this hyperparameter is deferred.
6.7 Detection results
BUTD-DETR is evaluated as an object detector using prompts containing all 18 ScanNet classes. Referential-data training and detection prompts consistently improve detection, although the model remains below 3DETR in the reported comparison.
- Detection setup: BUTD-DETR forms an 18-class detection prompt for each ScanNet scene to find all object instances.This evaluation treats object detection as grounding a prompt containing the benchmark classes.
- Detection performance: Training with detection prompts performs on par with the DETR+KPS+iter ablation but worse than 3DETR.The comparison uses BUTD-DETR trained on ScanNet and on a language-grounding benchmark.
- Detection performance: 1.7% improvement follows from adding a softmax loss over the 18 ScanNet classes.The paper notes that this fixed-class objective is unsuitable for language grounding, whose labels are not predefined or limited.
- Joint training: Up to 3.7% improvement on ScanNet follows from training on 3D referential datasets without access to additional scenes.The authors interpret this as evidence that co-training with grounding and detection prompts benefits both tasks.
6.8 More qualitative results
Qualitative examples show BUTD-DETR recovering missed referents and tightening predictions, while also exposing failures in proposal ranking, especially for complex, longer utterances.
- RefCOCO: BUTD-DETR predicts a chair missed by the detector, enabling grounding where proposal-bottlenecked approaches cannot.The figure also shows successful tight-box prediction when detector proposals have low IoU.
- RefCOCO: Low-IoU detector proposals can still lead BUTD-DETR to tight boxes around the referent object.These examples contrast detector proposals in blue with model predictions in green.
- SR3D: SR3D failures show that correctly ranking the target box over competing proposals remains difficult even when the box is present.The figure identifies this as a model failure rather than a detector-miss case.
- NR3D and SR3D: NR3D examples involve more complex, longer utterances, including a reported failure case.Additional qualitative results are provided for SR3D and NR3D.