Source-linked AI summary
3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence
Hao Tang, Ting Huang, Zeyu Zhang
TL;DR
3D captioning struggles with sparse, irregular point clouds, weak grounding, and limited OOD generalization across indoor and outdoor environments. 3D CoCa v2 unifies contrastive vision-language learning with caption generation and adds parameter-free test-time search; it improves reported in-domain and zero-shot OOD metrics over 3D CoCa.
Problem
3D captioning lacks robust spatial grounding and OOD generalization while operating on sparse, irregular point clouds across different environments.
Method
3D CoCa v2 combines a frozen CLIP-based semantic prior, spatially aware 3D encoding, multimodal contrastive-captioning objectives, and reward-guided test-time search over diverse candidates.
Results
+1.50 CIDEr@0.5 on ScanRefer, +1.61 CIDEr@0.5 on Nr3D, and +3.6 CIDEr@0.5 on zero-shot OOD TOD3Cap evaluation are reported over 3D CoCa.
Takeaways & Limitations
Inference-time search is presented as a practical plug-and-play direction for improving 3D caption quality and robustness without additional training or parameter updates.
Takeaways & Limitations
TTS adds inference latency and external judge-query costs, and its selection can fail when summaries are incomplete or judges overemphasize fluency over grounding.
Abstract
from arXiv · showhide
Spatial intelligence refers to the ability to perceive, reason about, and describe objects and their relationships within three-dimensional environments, forming a foundation for embodied perception and scene understanding. 3D captioning aims to describe 3D scenes in natural language; however, it remains challenging due to the sparsity and irregularity of point clouds and, more critically, the weak grounding and limited out-of-distribution (OOD) generalization of existing captioners across drastically different environments, including indoor and outdoor 3D scenes. To address this challenge, we propose 3D CoCa v2, a generalizable 3D captioning framework that unifies contrastive vision-language learning with 3D caption generation and further improves robustness via test-time search (TTS) without updating the captioner parameters. 3D CoCa v2 builds on a frozen CLIP-based semantic prior, a spatially-aware 3D scene encoder for geometry, and a multimodal decoder jointly optimized with contrastive and captioning objectives, avoiding external detectors or handcrafted proposals. At inference, TTS produces diverse caption candidates and performs reward-guided selection using a compact scene summary. Experiments show improvements over 3D CoCa of +1.50 CIDEr@0.5IoU on ScanRefer and +1.61 CIDEr@0.5IoU on Nr3D, and +3.8 CIDEr@0.25 in zero-shot OOD evaluation on TOD3Cap. Code will be released at https://github.com/AIGeeksGroup/3DCoCav2.
1 Introduction
3D captioning must handle sparse, irregular point clouds while grounding descriptions in spatial context and generalizing across environments. 3D CoCa v2 combines contrastive vision-language learning with caption generation and adds inference-time search to improve robustness under distribution shift.
- 3D captioning remains challenging because sparse, irregular point clouds and cluttered or partially observed objects require spatial-context reasoning.
- Existing pipelines use detection proposals or joint localization and description, but caption quality can remain coupled to detection accuracy and proposal noise.
- Test-Time Search generates diverse caption candidates and uses reward-guided selection from a compact scene summary without updating captioner parameters.
- 3D CoCa v2 unifies contrastive vision-language learning and 3D caption generation in a shared feature space, avoiding external detectors and handcrafted proposals.
- +1.50 CIDEr@0.5 on ScanRefer and +1.61 CIDEr@0.5 on Nr3D are reported with best-of-N Test-Time Search, while TOD3Cap zero-shot OOD evaluation yields +3.6 CIDEr@0.5 over 3D CoCa.
2 Related Work
Related work spans 3D captioning, contrastive 3D representation learning, and inference-time candidate selection. These directions motivate combining multimodal representation learning with search-based judging for 3D caption generation.
- 3D captioning localizes objects in 3D scenes and describes them in natural language, requiring geometric, relational, and linguistic understanding.
- Contrastive 3D methods learn transferable geometric representations, but they do not explicitly ground 3D representations in natural language.
- Inference-time search improves generation by sampling multiple candidates and selecting among them with auxiliary scoring signals, including language-model judges.
- 3D CoCa v2 applies this search paradigm to 3D captioning by generating candidates from a multimodal backbone and selecting a final caption with a scene-conditioned judge.
3 The Proposed Method
3D CoCa v2 unifies contrastive vision-language learning, 3D caption generation, and inference-time Test-Time Search in a shared framework. Its spatial encoder captures geometry and semantics, while TTS selects reward-supported captions from diverse candidates without updating captioner parameters.
- Framework Overview: The framework combines a 3D Scene Encoder, frozen Text Encoder, Contrastive Learning module, and MultiModal Fusion Decoder.The backbone follows a unified contrastive-generative design inspired by CLIP and CoCa.
- 3D Scene Encoder: The 3D Scene Encoder converts unstructured point clouds into latent tokens that capture geometric and semantic content using point processing, task tokens, and a frozen CLIP Vision Transformer.Point tokens are produced from local patches, while task tokens inject captioning context before CLIP encoding.
- Contrastive Learning: The model aligns scene and text features in a shared embedding space by bringing matched 3D-text pairs together and separating mismatched pairs.Learnable projection heads and L2 normalization prepare the features for cosine-similarity-based contrastive learning.
- Multimodal Decoder and Training: The multimodal decoder generates captions autoregressively by combining causal self-attention with cross-attention to the encoded scene representation.Training combines contrastive loss with standard cross-entropy captioning loss, while the global scene embedding conditions generation.
- Test-Time Search: Test-Time Search generates diverse caption candidates and uses a compact retrieved scene summary plus reward-guided judging to select the final caption.TTS is applied only during inference and introduces no additional trainable parameters or losses.
4 Experiments
Experiments evaluate 3D CoCa v2 on indoor benchmarks and zero-shot outdoor OOD data, including ablations and qualitative analyses. The method improves captioning quality and robustness, while TTS trades added latency for more grounded descriptions.
- Datasets and evaluation: 3D CoCa v2 is evaluated on ScanRefer and Nr3D for indoor captioning, and TOD3Cap for zero-shot OOD generalization.TOD3Cap evaluation uses oracle boxes to isolate captioning under distribution shift.
- OOD evaluation: 3D CoCa v2 achieves the best zero-shot TOD3Cap performance and consistently improves over 3D CoCa under indoor-to-outdoor distribution shift.All models are trained on indoor data only and evaluated without TOD3Cap fine-tuning.
- Qualitative results: Qualitative comparisons show more informative, fine-grained, and functionally grounded captions than 3D CoCa, especially in cluttered indoor scenes.TTS also favors specific object identities and layout cues over generic room-level descriptions.
- Qualitative results: TTS increases hallucination-aware caption faithfulness under OOD shifts, while stronger judge models generally improve CIDEr and reduce hallucination rates.GPT-5 provides the best reported trade-off among the evaluated judges.
5 Test-Time Efficiency
Test-Time Search improves caption selection at the cost of additional inference latency. Its overhead is concentrated in repeated decoding and judge scoring, and the trade-off can be adjusted through candidate count and judge choice.
- Latency trade-off: TTS raises per-scene latency from 0.55s to 1.78s, a 3.24× increase with N=8.The one-time backbone encoding remains 0.18s, while decode-plus-judge overhead is 1.60s.
- Latency trade-off: TTS adds inference cost through repeated candidate decoding and external LLM judge scoring, without changing backbone encoding cost.The quality-cost trade-off can be adjusted by N and the judge model.
6 Limitation and Future Work
3D CoCa v2’s test-time search introduces practical efficiency costs and remains constrained by the quality of its judging and scene summary. Future work targets more efficient inference, richer evidence representations, and broader 3D settings.
- Limitations: TTS increases inference-time latency and may add costs from best-of-N decoding and external judge queries.These overheads can be undesirable for strict real-time deployment, although N and judge choice offer a controllable quality-efficiency trade-off.
- Limitations: Judge-guided selection depends on judge reliability and a fixed scoring prompt, while incomplete summaries can cause failures.The judge may over-emphasize fluency over grounding when scene evidence is insufficient.
- Limitations: Lightweight scene summaries may miss fine-grained spatial relations or rare attributes, limiting penalties for subtle hallucinations.The authors propose more structured evidence representations to address this boundary.
- Future Work: Future work includes adaptive N, early stopping, lightweight judges, and extensions to outdoor LiDAR, dynamic scenes, and embodied scenarios.The proposed embodied settings would couple captioning with actions and long-horizon memory.
7 Conclusion
The paper presents 3D CoCa v2 as a unified contrastive-generative 3D captioner extended with inference-only test-time search. Experiments across indoor benchmarks and outdoor out-of-distribution scenes report improved caption quality and robustness, motivating inference-time search and stronger scene summarization.
- Conclusion: 3D CoCa v2 extends the 3D CoCa backbone with inference-only TTS using diverse candidates, an external LLM judge, and a compact scene summary.The approach requires no additional training or parameter updates.
- Conclusion: Experiments on in-domain indoor benchmarks and out-of-distribution outdoor scenes consistently enhance caption quality and robustness under distribution shift.The conclusion reports this pattern across both evaluation settings.
- Conclusion: The work motivates future research on stronger scene summarization, efficient judging, and generalizable 3D vision-language modeling for embodied applications.These directions are presented as practical extensions of inference-time search.