PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

Ting Huang, Zeyu Zhang, Hao Tang

arXiv:2507.23478v1cs.CV

TL;DR

Current 3D-VLMs face challenges in spatial reasoning, generalization, and fixed-view assumptions, despite the importance of 3D scene understanding for intelligent systems. 3D-R1 combines synthetic CoT cold-start data, RL training, and dynamic view selection. The authors report an average 10% improvement across various 3D scene benchmarks.

  • Problem

    Current 3D-VLMs face challenges in complex spatial reasoning and dynamic contexts, limited generalization from static training data, and reliance on predefined views.

    Source: Introduction · p. 2

  • Method

    3D-R1 cold-starts on Scene-30K synthetic CoT data, applies GRPO-based RL with format, perception, and semantic-similarity rewards, and dynamically selects views.

    Source: Introduction · p. 2

  • Results

    3D-R1 achieves an average improvement of 10% across various 3D scene benchmarks and performs strongly across diverse scene tasks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors position 3D-R1 as a generalist 3D VLM for unified scene understanding.

    Source: Conclusion · p. 8

    Scene-30K is primarily synthetic; response-level RL lacks temporally grounded feedback, and view selection assumes static scenes and a fixed candidate-view pool.

    Source: Limitation and Future Work · p. 8

Abstract

from arXiv · show

Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. However, current 3D VLMs often struggle with robust reasoning and generalization due to limitations in high-quality spatial data and the static nature of viewpoint assumptions. To address these challenges, we propose 3D-R1, a foundation model that enhances the reasoning capabilities of 3D VLMs. Specifically, we first construct a high-quality synthetic dataset with CoT, named Scene-30K, leveraging existing 3D-VL datasets and a data engine based on Gemini 2.5 Pro. It serves as cold-start initialization data for 3D-R1. Moreover, we leverage RLHF policy such as GRPO in the reinforcement learning training process to enhance reasoning capabilities and introduce three reward functions: a perception reward, a semantic similarity reward and a format reward to maintain detection accuracy and answer semantic precision. Furthermore, we introduce a dynamic view selection strategy that adaptively chooses the most informative perspectives for 3D scene understanding. Extensive experiments demonstrate that 3D-R1 delivers an average improvement of 10% across various 3D scene benchmarks, highlighting its effectiveness in enhancing reasoning and generalization in 3D scene understanding. Code: https://github.com/AIGeeksGroup/3D-R1. Website: https://aigeeksgroup.github.io/3D-R1.

Introduction

The introduction identifies spatial reasoning, generalization, and fixed-view assumptions as challenges for 3D-VLMs, then presents 3D-R1’s CoT initialization, RL training, and dynamic view selection. It reports an average 10% improvement across 3D scene benchmarks.

Introduction

  • The introduction situates 3D scene understanding across tasks including dense captioning, object captioning, question answering, dialogue, grounding, reasoning, and planning.The authors connect spatial understanding to applications such as robotics and mixed reality.Source: Introduction · p. 1
  • Current 3D-VLMs struggle with complex spatial relationships and dynamic contexts; SFT generalization and fixed-view assumptions are also limitations.Predefined views can introduce irrelevant visual content or miss details occluded from those views.Source: Introduction · p. 2
  • The authors motivate RL for 3D-VLM reasoning by asking whether reinforcement learning can improve reasoning ability in these models.They cite DeepSeek-R1’s use of RL to induce complex cognitive reasoning in LLMs.Source: Introduction · p. 2
  • 3D-R1 combines a 30K-sample synthetic CoT dataset for cold-start initialization with GRPO-based RL and dynamic selection of informative scene views.Its rewards target format, perception, and semantic similarity; view ranking uses learned relevance scores.Source: Introduction · p. 2
  • 3D-R1 achieves an average improvement of 10% across various 3D scene benchmarks.The introduction also reports strong performance across diverse 3D scene tasks.Source: Introduction · p. 2

Related Work

Related work traces 3D vision-language research from object-level models to scene-level systems and surveys task-specific work in captioning, question answering, grounding, and RL-enhanced reasoning.

3D scene understanding

  • Dense captioning work spans object-level detection-and-captioning pipelines and scene-level descriptions, including transformer and LLM-augmented approaches.Scan2Cap introduced object captioning; Vote2Cap-DETR variants use an encoder–decoder, while 3D-CoCa and LSceneLLM address scene descriptions.Source: Related Work · pp. 2–3
  • 3D-QA benchmarks test spatial reasoning, while visual-grounding research aligns language expressions with objects in 3D scenes.SQA3D adds an embodied-agent viewpoint; reviewed grounding approaches include cross-modal transformers and GPT-4-based zero-shot grounding.Source: Related Work · p. 3

Reinforcement learning

  • Recent RL-based work applies graph-centric rewards to scene-graph prediction and guides multimodal LLMs in fine-grained visual inspection.The surveyed systems are R1-SGG and DIP-R1.Source: Related Work · p. 3

3D vision language models

  • 3D-VLMs progressed from object point-cloud models to scene-level systems that process whole rooms and can produce structured spatial outputs.PointLLM answers questions about object shape and attributes, while LLaVA-3D and 3D-LLaVA address whole scenes.Source: Related Work · p. 3

The Proposed Method

3D-R1 combines a Scene-30K cold start with reinforcement learning and dynamic view selection. Its method also includes a CoT data engine, reward-guided policy updates, and dense-captioning comparisons on ScanRefer and Nr3D.

Overview

  • The overview combines Scene-30K cold-start initialization, GRPO-based reinforcement learning, and adaptive selection of informative scene views.The view-selection method scores candidate views to focus on relevant spatial context.Source: Overview · p. 4

CoT Data Engine

  • The CoT data engine uses scene descriptions and Gemini 2.5 Pro to generate and filter examples, yielding 30K examples for cold-start training.The pipeline first produces roughly 35K examples, then filters for missing structure or inconsistent reasoning.Source: CoT Data Engine · p. 4

Cold Start Stage

  • End-to-end reinforcement learning proved unstable, so supervised fine-tuning on Scene-30K teaches structured reasoning and answer formats before policy optimization.Pure RL often failed to produce coherent CoT sequences and semantically aligned answers.Source: Cold Start Stage · p. 4

Reinforcement Learning

  • GRPO samples multiple responses and combines grouped rewards for format adherence, bounding-box IoU, and semantic CLIP similarity to update the policy.The perception reward evaluates spatial precision, while the semantic reward compares answer embeddings.Source: Figure 4 · p. 5, Reinforcement Learning · p. 5
  • 3D-R1 surpasses previous 3D specialists on both ScanRefer and Nr3D for dense captioning.Table 2 compares methods trained with standard per-word cross-entropy and no additional 3D scenes.Source: Table 2 · p. 6

Dynamic View Selection

  • Dynamic view selection scores candidate views using complementary signals, then learns their fusion weights to produce a utility score for each view.L2 regularization on the text-relevance weight discourages overly dominant text influence.Source: Dynamic View Selection · p. 6

Experiment

The experiments evaluate 3D-R1 across seven 3D scene-understanding tasks using standard datasets and task-specific metrics. The reported results show improvements over prior methods across the evaluated tasks.

Datasets and Metrics

  • The evaluation uses ScanRefer and Nr3D for 3D dense captioning and visual grounding, Cap3D for object captioning, ScanQA for question answering, 3D-LLM for dialogue and planning, and SQA3D for reasoning.Metrics include Acc@sIoU for visual grounding, human and CLIP-based retrieval measures for object captioning, and CIDEr, BLEU, METEOR, and ROUGE-L for several text-generation tasks.Source: Datasets and Metrics · pp. 6–7, Main Results · p. 7

Main Results

  • 3D-R1 consistently outperforms existing methods on the ScanQA validation set and two test benchmarks for 3D question answering.Source: Table 3 · p. 7, Main Results · p. 7
  • 3D-R1 consistently outperforms existing methods on ScanRefer and Nr3D for 3D scene dense captioning.The task requires localizing and describing objects in a 3D scene.Source: Main Results · p. 7
  • 3D-R1 achieves the highest scores across all evaluation criteria on Cap3D object captioning.The comparisons include DiffuRank trained with all 28 views, 6 horizontal views, or 6 bottom views.Source: Main Results · p. 7, Table 4 · p. 8
  • 3D-R1 achieves the highest scores across all reported metrics on SQA3D for 3D reasoning.Source: Main Results · p. 7, Table 5 · p. 8
  • On 3D-LLM, 3D-R1 achieves state-of-the-art results across all evaluation metrics for 3D dialogue.Source: Main Results · p. 7, Table 5 · p. 8
  • 3D-R1 consistently outperforms existing methods on Nr3D and ScanRefer for 3D visual grounding.Source: Main Results · p. 7, Table 6 · p. 8
  • On 3D-LLM, 3D-R1 surpasses all baselines across all evaluation criteria for 3D planning.Planning generates sequential action plans from instructions and 3D contextual understanding.Source: Main Results · pp. 7–8, Table 5 · p. 8

Limitation and Future Work

The authors identify limitations in synthetic CoT supervision, response-level RLHF feedback, and the static-scene assumptions of dynamic view selection. They propose embodied real-world applications and a world model as future directions.

Limitation and Future Work

  • Scene-30K is primarily synthetic and may not capture the diversity and ambiguity of real-world human reasoning.Source: Limitation and Future Work · p. 8
  • GRPO-based RLHF operates at the response level without temporally grounded feedback, limiting long-range reasoning and action in embodied settings.Source: Limitation and Future Work · p. 8
  • Dynamic view selection assumes static scenes and a fixed candidate-view pool, which may restrict real-time interactive use; future work proposes real-world embodied AI and a world model for predicting future scene states.Source: Limitation and Future Work · p. 8

Conclusion

3D-R1 combines Scene-30K cold-start supervision, GRPO-based reward optimization, and dynamic view selection. Evaluations across seven representative benchmarks report improvements over prior methods.

Conclusion

  • 3D-R1 combines cold-start initialization, GRPO-based reinforcement learning with perception, semantic-similarity, and format rewards, and task-relevant multiview ranking.Source: Conclusion · p. 8
  • Scene-30K is a high-quality 30K-example chain-of-thought dataset used for cold-start initialization of 3D vision-language models.Source: Conclusion · p. 8
  • Across seven representative 3D benchmarks, 3D-R1 achieves significant improvements over prior methods.Source: Conclusion · p. 8

Ablation Study

The ablations assess reinforcement-learning rewards, dynamic view selection and its scoring weights, modality encoders, and LoRA rank. Together, they report gains from combined rewards and adaptive views, with full multimodal inputs and rank δ = 12 favored in their respective comparisons.

Reinforcement learning

  • Combined GRPO rewards raise ScanQA CIDEr from 97.95 to 106.45 and ScanRefer C@0.25 from 85.20 to 91.85 over SFT.The format, perception, and semantic rewards target output consistency, spatial grounding, and meaning alignment, respectively.Source: Ablation Study · p. 13, Table 7 · p. 13

Dynamic view selection

  • Learned six-view selection raises 3D object captioning CLIP R@1 from 30.18 with fixed horizontal six views to 32.23 and also improves visual grounding.The comparison covers learned selection against all-views, horizontal six-views, and bottom six-views baselines.Source: Ablation Study · p. 13, Table 8 · p. 13
  • Balanced view-score weights achieve the best reported performance across tasks, with wt = 0.3, wc = 0.5, and wclip = 0.5.The reported performance landscape identifies wt around 0.3–0.4 with balanced visual cues as optimal.Source: Ablation Study · p. 13, Table 9 · p. 13, Figure 5 · p. 14

Architecture and hyperparameters

  • Adding depth and then point-cloud encoders improves SQA3D reasoning and 3D-LLM planning, with the full 3D-R1 model best across all metrics.The ablation starts from text and image encoders; the table caption describes each subsequent row as adding one encoder.Source: Ablation Study · p. 13
  • Increasing LoRA rank δ from 4 to 12 raises ScanQA CIDEr from 94.57 to 106.45 and Nr3D accuracy from 63.12 to 68.80; gains beyond 12 are marginal.The authors identify δ = 12 as the best performance-efficiency trade-off.Source: Ablation Study · pp. 13–14

Implementations Details

Implementation uses a filtered synthetic chain-of-thought dataset for initialization, a multimodal architecture with parameter-efficient tuning, and GRPO training. A learned utility score selects six scene views during training and inference.

Implementations Details

  • Scene-30K filtering reduces 35,248 raw CoT examples to 30,012 retained examples for cold-start initialization.Criteria include valid tagged output, minimum lengths, at least three reasoning steps, a final reference to the target, logical consistency, and answer similarity of at least 0.8.Source: Implementations Details · p. 14
  • The model combines projected image, depth, and point-cloud features with text tokens, while freezing the base VLM backbone and tuning adapters and modality encoders.The base is Qwen2.5-VL-7B-Instruct; the encoders are SigLIP-2, Depth-Anything v2, and Point Transformer v3.Source: Implementations Details · p. 14

continue

  • LoRA adapters in the final eight of 28 transformer blocks use rank δ = 12 and add about 12M trainable parameters, approximately 0.17% of the backbone.About 142M parameters are updated in total, compared with about 7B for full fine-tuning, a reported 98% reduction in trainable parameters.Source: continue · p. 15
  • Training applies two SFT epochs on Scene-30K followed by two GRPO epochs, with a KL penalty of β = 0.02 against the frozen SFT model.SFT uses AdamW; the RL stage uses Adam at a fixed learning rate of 10^-6.Source: continue · p. 15
  • At training and inference, the learned utility score ranks scene views using text relevance, spatial coverage, and CLIP similarity, then selects the top six with corresponding depth inputs.Visual features for each view are extracted using a pretrained SigLIP-2 encoder.Source: continue · p. 15

Visualization

The paper presents visualizations to qualitatively assess 3D-R1 across scene-understanding tasks. The examples are described as highlighting reasoning, spatial comprehension, and multimodal alignment.

Visualization

  • Visualizations across 3D scene tasks are presented as qualitative evidence of reasoning ability, spatial comprehension, and multimodal alignment.The passage identifies these as visual assessments rather than quantitative results.Source: Visualization · p. 15

Response

The response section provides example Scene-30K chain-of-thought data and qualitative 3D-R1 outputs across several scene-understanding tasks. The captions identify the task represented by each result figure.

Response

Loading 2507.23478v1…