Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy

1 INTRODUCTION

• Existing 3D vision–language methods often depend on complex 3D inputs and external depth, pose, or other modules, limiting scalability and transferability.These pipelines can add data and engineering overhead, memory use, and latency.Source: §1 · p. 2
• Reconstruction and reasoning are described as interdependent: geometry supports semantic understanding, while semantic reasoning provides context for geometric modeling.This motivates their integration in Vid-LLM.Source: §1 · p. 2
• Vid-LLM jointly reconstructs scenes and performs 3D vision–language reasoning from monocular video, without relying on dense 3D inputs or prior poses.The framework is presented as practical for real-world deployment.Source: §1 · p. 2
• Vid-LLM uses a Cross-Task Adapter to align 3D geometry priors with vision–language representations, alongside a two-stage training strategy.The stated aims are improved integration, stability, and performance.Source: §1 · p. 2
• Experiments report superior performance in question answering, dense captioning, and visual grounding on real datasets.The paper presents these evaluations as evidence of the method's effectiveness.Source: §1 · p. 2

3D-aware Multimodal Large Language Models

• Existing 3D-MLLMs commonly use explicit geometric inputs, such as point clouds or reconstructed scenes, which impose acquisition, preprocessing, and computation burdens.VGLLM is noted as a video-based 3D-MLLM that extracts geometry features from video using VGGT's geometry encoder.Source: §2 · p. 3

3D Reconstruction

• Reconstruction methods have progressed from multi-view pipelines to feed-forward video models, but their integration with vision–language semantic reasoning remains weakly explored.The cited approaches predict depth, pose, and point clouds directly from video inputs.Source: §2 · p. 3

Geometry Priors in Vision-Language Models

• Geometry-prior methods range from explicit input injection to training-time internalization and modular integration; Vid-LLM instead reconstructs geometry from video and aligns it with a semantic branch.The categories differ in whether geometry is supplied as input, embedded through data and training, or added through modules or foundation-model outputs.Source: §2 · p. 3

3.1 CROSS-TASK ADAPTER

• The Cross-Task Adapter aligns 3D geometry priors with vision–language representations using bridge tokens that exchange complementary geometric and semantic information.The resulting task-specific features pass to the reconstruction and reasoning branches.Source: §3.1 · pp. 3–4

3.2 3D RECONSTRUCTION MODEL

• The reconstruction branch estimates camera parameters and relative depth, while the Metric Depth Model supplies global scale cues for real-scale alignment.A scene-level scale factor is the median of factors computed from 16 randomly sampled images; the DPT depth head is retained for its more accurate texture details.Source: §3.2 · pp. 4–5

3.4 TRAINING STRATEGY

• Stage 1 distills geometric and semantic representations into the shared encoder and adapter, and Stage 2 jointly optimizes reconstruction and 3D vision–language tasks.The geometry and semantic teachers are pretrained DINO and CLIP encoders; auxiliary reconstruction losses support reconstruction capability and real-scale consistency.Source: §3.4 · pp. 5–6
• During joint training, reconstruction and 3D vision–language losses optimize the shared adapter, but reasoning losses do not optimize the reconstruction model.The paper describes this one-way gradient flow as keeping the adapter a stable bridge for geometry–semantics interaction.Source: §3.4 · p. 6

4.2 3D VISION–LANGUAGE REASONING

• Vid-LLM beats the second-best video-based model by an average 11% on ScanQA and SQA3D, leads all reported grounding metrics, and scores 28.7 M@0.5 on Scan2Cap versus 29.0 for the best 3D-based model.The Scan2Cap n-gram metrics are governed by IoU@0.5.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 8

4.1 EXPERIMENTAL SETUP

• Evaluation follows dataset-specific standard protocols, uses consistent training data and settings across models, and computes ScanNet reconstruction metrics in metric scale.Training uses DINOv2-L and uniformly samples 32 frames per scene.Source: §4.1 · p. 7

4.3 COMPARISON WITH JOINT RECONSTRUCTION–REASONING MODELS

• Vid-LLM achieves the best overall joint-model performance, with 47.6% average relative 3D VL improvement over Uni3DR² and 12.1% over VGLLM-Rec across ScanQA, Scan2Cap, and ScanRefer.Its ScanNet reconstruction score is 0.582 versus Uni3DR²’s 0.580 despite Uni3DR² using ground-truth camera poses; inference takes 1.6–2.1 s/scene versus 2.7 s/scene for concatenation pipelines.Source: Table 5 · p. 9, §4.3 · p. 9

4.4 ABLATION STUDIES

• The full CTA with bridge tokens improves cross-task performance, with 16 tokens balancing accuracy and model complexity; metric depth combined with scale alignment yields the most accurate reconstruction and more reliable 3D VL results.Without metric depth supervision, scale ambiguity leads to near failure of 3D VL reasoning.Source: §4.4 · p. 10, Table 6 · p. 10
• Dual-teacher supervision with relational consistency achieves the best training performance, while single-stage training converges slowly and stabilizes at higher loss.Removing relational loss further slows convergence and reduces accuracy.Source: §4.4 · p. 10, Figure 5 · p. 10

5 CONCLUSION

• Vid-LLM achieves state-of-the-art results on several 3D vision-language benchmarks and remains competitive on others.Ablation studies validate the effectiveness of each component.Source: §5 · p. 10
• The Cross-Task Adapter aligns 3D geometry priors with vision-language representations, improving their integration and reasoning robustness under uncertain geometry.The compact video-based architecture extracts geometric cues from video and feeds them into the LLM through 3D patch construction.Source: §5 · p. 10
• A two-stage training strategy improves training stability and convergence speed.Source: §5 · p. 10

6 REPRODUCIBILITY STATEMENT

• Part of the implementation is provided in the Supplementary Materials.Source: §6 · p. 11
• The authors state that complete code, datasets, and pretrained checkpoints will be released upon acceptance.Source: §6 · p. 11
• The planned release is intended to enable full verification of the results.Source: §6 · p. 11

A APPENDIX

• Appendix A.1 contains qualitative results and challenging cases for Vid-LLM.Source: A APPENDIX · p. 17
• Appendix A.2 contains supplementary experiments on reconstruction performance.Source: A APPENDIX · p. 17
• Appendix A.3 contains a statement about the use of large language models.Source: A APPENDIX · p. 17

A.1 QUALITATIVE RESULTS AND CHALLENGING CASES

• Figure 6 illustrates ScanNet reconstruction alongside 3D question answering, dense captioning, and visual grounding.The examples cover questions about object counts, locations, and attributes; detailed room and object descriptions; and localization from textual instructions.Source: §A.1 · p. 17, Figure 6 · p. 17
• Limited viewpoints, reflective surfaces, and oblique views can leave monocular reconstructions incomplete, affecting instance counting and 3D box extents.In the examples, a partly observed chair is missing from reconstruction, and a reflective tabletop yields a predicted box smaller than ground truth.Source: §A.1 · p. 17, Figure 7 · p. 18
• Vid-LLM produces an accurate cabinet caption despite a poorly reconstructed top surface when rich 2D semantic cues are available.The Cross-Task Adapter and bridge-token mechanism jointly align semantic and geometric information.Source: §A.1 · p. 17, Figure 7 · p. 18
• The appendix also reports camera-pose evaluation on Co3Dv2 and RealEstate10K, and depth and reconstruction results on NYU Depth v2 and ScanNet.For the depth and reconstruction evaluations, relative-scale outputs are aligned to ground-truth metric scale before evaluation.Source: Table 7 · p. 18, Table 8 · p. 19, Table 9 · p. 19

• Evaluation covers camera pose estimation on Co3Dv2 and RealEstate10K, depth prediction on NYU Depth v2, and point-cloud reconstruction on ScanNet.Depth and reconstruction require real-scale outputs, while pose estimation is scale-invariant.Source: §A.2 · p. 19
• The reconstruction comparisons span geometry-based, learning-based, pose-dependent, and pose-free methods; VGGT is both a reconstruction benchmark and the framework’s teacher.Source: §A.2 · p. 19
• In the end-to-end real-scale setting, Vid-LLM has lower log10 error on NYU and higher F-score and accuracy on ScanNet than VGGT, and surpasses other real-scale-output models.Source: §A.2 · p. 20
• Vid-LLM is close to VGGT on pose estimation on Co3Dv2 and RealEstate10K, and exceeds it on Co3Dv2 RTA@15.Source: §A.2 · pp. 19–20

• The authors used LLMs to polish manuscript language.Source: §A.3 · p. 20
• They report verifying the assisted text and attribute the paper’s intellectual contributions, ideas, and conclusions to themselves.Source: §A.3 · p. 20