3D CoCa: Contrastive Learners are 3D Captioners

1 Introduction

• Sparse point clouds, cluttered object distributions, difficult spatial relations, and weak visual-textual alignment make 3D scene captioning challenging.Existing methods treat vision and language as separate stages, resulting in suboptimal alignment between their representations.Source: §1 · pp. 1–2
• Two-stage detect-then-describe methods can require redundant-box suppression and make caption generation dependent on detection quality.The introduction motivates a unified approach that jointly learns spatially grounded captions and visual-text alignments using large-scale vision-language priors.Source: §1 · pp. 1–2
• 3D CoCa jointly learns contrastive 3D-text representations and caption generation, eliminating the need for external 3D object detectors.The model uses a shared feature space and a spatially aware encoder and decoder to align scene representations with language.Source: §1 · p. 2
• 52.84% C@0.5 on Nr3D and 77.13% C@0.5 on Scanrefer; 3D CoCa achieves state-of-the-art captioning performance on both benchmarks.The introduction reports these benchmark results as extensive evaluation outcomes.Source: §1 · p. 2

3D Dense Captioning

• Early dense captioning methods added spatial reasoning and higher-order relations, while transformer-based approaches captured geometric context or transferred semantics from 2D models.3DJCG jointly optimizes captioning and visual grounding, and UniT3D pretrains on point-cloud–text pairs.Source: §2 · p. 2
• Two-stage methods separate detection from description, whereas Vote2Cap-DETR and Vote2Cap-DETR++ jointly localize and caption objects in one stage.The two-stage separation can propagate errors and misalign vision and language; the one-stage models use learnable queries and iterative refinement.Source: §2 · p. 2

3D Pre-training and Vision-Language Foundations

• 3D geometric pretraining learns geometric features without natural-language grounding, while 3D vision-language pretraining aligns point-cloud segments with text.UniT3D is also described as learning multi-task 3D understanding from large-scale point-cloud–caption pairs.Source: §2 · pp. 2–3

Multimodal Large Language Models for 3D Scenes

• LLM-based 3D systems support interactive tasks, but typically rely on external language models and complex alignment procedures.The paper notes that these frameworks treat captioning as one of many tasks rather than a dedicated end-to-end objective.Source: §2 · p. 3

3D Dense Captioning

• 3D CoCa jointly trains a 3D scene encoder and text decoder with a contrastive vision-language objective, avoiding separate detection modules and post-hoc LLM integration.The paper presents this as its unified approach to 3D dense captioning.Source: §2 · p. 3

3.1 Overview

• 3D CoCa combines a 3D scene encoder, text encoder, contrastive learning module, and multimodal fusion decoder for captioning.The framework is inspired by contrastive alignment and multimodal captioning.Source: §3.1 · p. 3
• The framework adapts knowledge from large-scale 2D-text pretraining to point-cloud data and keeps most CLIP pretrained weights frozen.The paper says this preserves visual and linguistic representations while adding minimal parameters for 3D processing.Source: §3.1 · pp. 3–4

3.2 3D Scene Encoder

• The scene encoder tokenizes point clouds into local patches, encodes their geometry and appearance, and uses sampled patches to balance local detail with scene coverage.Farthest point sampling selects patch centers, and each center is grouped with its nearest neighbors before point-wise MLP encoding.Source: §3.2 · p. 4

3.2.2 Task token mechanism.

• Learnable task tokens are prepended to point tokens and attend over the point cloud to gather global semantic information relevant to caption generation.The paper describes them as a shared contextual bias that helps the encoder emphasize language-relevant scene elements.Source: §3.2.2 · p. 4, §3.2.3 · p. 4

3.2.3 Frozen CLIP vision encoder.

• The frozen CLIP vision Transformer processes the combined sequence and outputs latent features from which a task-conditioned global scene representation is derived for text alignment.Freezing the backbone preserves pretrained representations and reduces memory use and training time.Source: §3.2.3 · p. 4

3.3 Text Encoder

• The frozen CLIP text encoder maps natural-language descriptions into a compatible embedding space for alignment with 3D scene features.Its pretrained language knowledge is retained, and freezing is described as mitigating overfitting on relatively small 3D captioning datasets.Source: §3.3 · p. 4, §3.3.2 · p. 5

3.3.1 Text tokenizer.

• Captions are tokenized into ordered text embeddings with positional encodings, while subword units handle out-of-vocabulary words.The sequence is passed through the CLIP text Transformer.Source: §3.3.1 · p. 4, §3.3.2 · p. 4

3.3.2 Frozen CLIP text encoder.

• The final Transformer layer’s [CLS] output is the global caption representation used to align with the 3D scene feature.Keeping the CLIP text encoder fixed places this representation in a language embedding space directly comparable to CLIP visual features.Source: §3.3.2 · p. 5

3.4 Contrastive Learning Paradigm

• The contrastive strategy projects 3D scene and text features into a shared latent space, bringing corresponding scene-caption pairs closer and separating non-matching pairs.This follows the CLIP multimodal training objective.Source: §3.4 · p. 5

3.4.1 Feature alignment.

• Two learnable two-layer MLPs project scene and text features to a common dimension, then L2 normalization enables cosine-similarity comparison.The learned projections are tuned for alignment.Source: §3.4.1 · p. 5

3.4.2 Contrastive loss function.

• The InfoNCE loss maximizes each scene’s similarity to its matched caption and minimizes similarity to unmatched captions in the batch.A learnable temperature scales the logits before softmax.Source: §3.4.2 · p. 5

3.5 Multi-Modal Fusion Decoder

• An autoregressive Transformer decoder generates scene descriptions word by word while using cross-attention to incorporate 3D scene context.It fuses aligned representations to produce fluent, contextually grounded sentences.Source: §3.5 · p. 5

3.5.1 Cross-Attention mechanism.

• Decoder queries attend to encoded 3D scene tokens, producing weighted visual information that supports captions reflecting scene context and local details.Self-attention and cross-attention are interleaved to combine textual and visual information while preserving sequential coherence.Source: §3.5.1 · pp. 5–6

3.5.2 Training objectives and joint optimization.

• Captioning loss and contrastive loss are jointly optimized, with the total objective balancing alignment against natural-language generation.The captioning loss compares generated text with the reference; the paper sets the balancing weight to give roughly equal importance to both objectives. The joint scheme is described as refining caption-relevant visual features and cross-modal correspondence.Source: §3.5.2 · p. 6

4.1.1 Datasets.

• ScanRefer contains 36,665 descriptions of 7,875 objects in 562 scenes, while Nr3D contains 32,919 descriptions of 4,664 objects in 511 scenes.The training data come from ScanNet’s 1,201 scenes; evaluation uses 9,508 ScanRefer descriptions and 8,584 Nr3D descriptions from the validation set.Source: §4.1.1 · p. 6

4.1.2 Evaluation metrics.

• Model performance is evaluated with CIDEr, BLEU-4, METEOR, and ROUGE-L; caption-generation evaluation uses m@kIoU with IoU thresholds set to 0.25 and 0.5.Source: §4.1.2 · pp. 6–7

4.2 Implementation Details

• Models are trained for 1,080 epochs with cross-entropy and contrastive losses, using AdamW, a batch size of 4, and a single RTX4090 GPU.The scene encoder is EPCL integrated with a frozen CLIP visual encoder, and text embeddings come from a frozen CLIP text encoder.Source: §4.2 · p. 7

4.3 Comparative Study

• The comparative study reports C, M, B-4, and R for ScanRefer at IoU 0.25 and 0.5 and Nr3D at IoU 0.5; dashes mark unavailable results.C, M, B-4, and R denote CIDEr, METEOR, BLEU-4, and ROUGE-L, respectively.Source: §4.3 · p. 7

4.3.1 Scanrefer.

• 3D CoCa outperforms existing methods across ScanRefer data settings and IoU thresholds, and improves caption generation on Nr3D.Table 1 covers ScanRefer with and without additional 2D input at IoU 0.25 and 0.5; Table 2 evaluates Nr3D at IoU 0.5.Source: §4.3.1 · p. 7, §4.3.2 · p. 7, Table 1 · p. 7

4.5 Qualitative Results

• Figure 3 compares descriptions from 3D CoCa and Vote2Cap-DETR++ with the ground-truth captions on ScanRefer scenes.The authors report that 3D CoCa accurately describes scene attributes and categories.Source: Figure 3 · p. 8, §4.5 · p. 8

4.4.1 Contrastive learning loss impact analysis.

• CIDEr rises from 74.12% at λ=0 to 79.55% at λ=0.5, peaks at λ=1, and drops slightly at λ=2.0 while remaining above the no-contrastive result.The reported analysis says moderate contrastive loss improves all indicators and supports alignment and scene semantics.Source: §4.4.1 · pp. 7–8, Table 3 · p. 7
• EPCL with CLIP features exceeds PointNet++ by 12.94% CIDEr under the same settings.The authors report improvements across indicators and describe stronger semantic expression and spatial modeling with EPCL and pretrained CLIP visual features.Source: §4.4.2 · p. 8, Table 4 · p. 8

4.4.2 Point cloud encoder structure analysis.

• Replacing the CoCa-style multimodal decoder with GPT-2 reduces description quality when visual encoder features are held constant.The authors attribute the difference to the CoCa-style decoder incorporating aligned visual features more effectively into language generation.Source: §4.4.3 · p. 8, Table 5 · p. 8

• 3D CoCa unifies contrastive 3D-text representation learning and caption generation within a single model.The framework eliminates explicit 3D object detectors or proposal stages.Source: §5 · p. 9
• The shared feature space aligns 3D and text representations, which the conclusion associates with improved spatial reasoning and more precise semantic grounding.Source: §5 · p. 9
• Experiments on two widely used datasets report that 3D CoCa significantly outperforms existing methods across standard captioning metrics.Source: §5 · p. 9