PaperlayerView exact revision on arXiv

Source-linked AI summary

SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun, Hehe Fan

arXiv:2608.05137v3cs.CV

TL;DR

Existing 3D multimodal models often use fixed modality combinations even though different queries favor different sensory cues, which can introduce irrelevant information and waste computation. SmartMage dynamically selects modalities with SMART and guides modality-specific expert activation with MAGE, achieving state-of-the-art results across five 3D benchmarks while remaining competitive with RGB-only inputs.

  • Problem

    Fixed modality combinations overlook query-dependent modality relevance, allowing irrelevant inputs to introduce semantic noise while informative modalities receive insufficient attention.

  • Method

    SmartMage dynamically orchestrates heterogeneous modalities through SMART routing and MAGE's modality-aware expert specialization.

  • Results

    SmartMage achieves state-of-the-art performance on five standard 3D scene understanding benchmarks and remains competitive on RGB-only video benchmarks.

  • Takeaways & Limitations

    Dynamic routing better exploits complementary multimodal cues, with adaptive selection achieving the best performance across the reported ablation benchmarks.

  • Takeaways & Limitations

    Token-budget-driven view and point-cloud reductions may omit critical viewpoints and fine-grained details, potentially degrading performance when complementary modalities are insufficient.

Abstract

from arXiv · show

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTing (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.

1 Introduction

SmartMage addresses query-dependent modality needs in 3D scene understanding by dynamically selecting relevant modalities and assigning them to specialized experts. It combines semantic-guided routing with modality-aware expert specialization to improve multimodal reasoning efficiency and performance.

  • 3D scene understanding combines complementary appearance, geometry, structure, and spatial-layout information from RGB-D video, point clouds, voxels, and BEV projections.
  • Different questions favor different modalities, so indiscriminately combining all inputs can add semantic noise, waste computation, and dilute reasoning.Geometric questions benefit more from point clouds, whereas appearance-related questions primarily rely on RGB inputs.
  • SmartMage decomposes multimodal reasoning into selecting relevant modalities and assigning modality tokens to specialized experts through SMART and MAGE.
  • SMART selects query-relevant auxiliary modalities by jointly using semantic priors, text–modality similarity, and modality quality while retaining RGB as the fixed primary modality.
  • MAGE predicts modality attribution and incorporates modality–expert affinity priors into sparse MoE gating to promote specialization across modalities.
  • SmartMage achieves state-of-the-art performance on five 3D benchmarks, including gains of +5.1 Acc@0.5 on ScanRefer and +6.4 F1@0.5 on Multi3DRefer over Ross3D.It also remains highly competitive on three RGB-only video benchmarks despite missing 3D inputs.

2 Related Work

3D scene understanding methods have progressed from single explicit 3D representations toward multimodal and 2D-video-based approaches. However, competitive spatial reasoning from temporal visual inputs leaves open how heterogeneous modalities should be integrated.

  • Early methods primarily use point clouds or voxels for LLM-based 3D grounding and question answering.
  • Later approaches project 2D features into 3D space to improve fine-grained understanding.
  • Recent 2D video-based methods achieve competitive performance against explicit-3D methods, suggesting spatial reasoning can emerge from temporal visual inputs.

3 SmartMage

SmartMage dynamically routes heterogeneous 3D modalities according to query semantics, modality alignment, and quality, then guides token-level expert specialization. Its end-to-end training jointly optimizes semantic routing and modality-consistent expert assignment.

  • SmartMage: SmartMage processes RGB, depth, BEV, point-cloud, and voxel inputs through a unified embedding pipeline before global SMART selection and local MAGE specialization.The architecture follows a global-to-local sequence: heterogeneous feature extraction, input-level modality selection, and internal expert routing.
  • Semantic-guided Modality Adaptive Routing: SMART selects RGB as the primary modality and supplements it with query-relevant modalities using semantic priors, text–visual similarity, and modality quality.The modality quality evaluator derives reliability from activation strength, sparsity, and stability, enabling degraded modalities to be down-weighted.
  • Semantic-guided Modality Adaptive Routing: The SMART router combines modality-wise prior, semantic similarity, and quality scores into routing logits, with an RGB evidence gate adapting RGB reliance to available evidence.The three cues represent instruction demands, modality alignment, and modality reliability.
  • Modality-Aware Gating Expert: MAGE predicts token-level modality attribution and uses modality–expert affinity priors to bias sparse MoE gating toward modality-consistent expert specialization.Tokens are routed to top-k experts, while calibration encourages routing decisions to align with modality-aware priors.
  • Network Optimization: End-to-end selective fine-tuning freezes visual encoders, trains lightweight adapters, and updates selected router–expert branches using eight experts with top-2 routing.The overall objective combines language modeling, semantic alignment and discrimination, modality attribution, expert calibration, and expert balancing losses.

4 Experiments

SmartMage is evaluated across standard 3D, RGB-only video, and diagnostic settings, with results showing strong benchmark performance, adaptive modality preferences, and efficiency benefits. Ablations and controlled comparisons support the value of semantic routing and modality-aware expert allocation.

  • Experimental Setup: SmartMage evaluates five 3D benchmarks, RGB-only video benchmarks, and ScanFacet, a diagnostic benchmark organized by semantic question categories.The evaluation also uses standardized training and implementation settings described for the unified corpus and model optimization.
  • Comparison with State-of-the-art Methods: SmartMage consistently outperforms prior methods across five 3D benchmarks spanning question answering, dense captioning, and visual grounding.Reported results include 32.6 EM@1 on ScanQA and 66.8 EM@1 on SQA3D, with additional gains on Scan2Cap, ScanRefer, and Multi3DRefer.
  • Controlled Comparison: Under matched modalities, resolution, and backbone, SmartMage outperforms Ross3D across all five benchmarks, separating the comparison from input or backbone advantages.The matched setting uses RGB, BEV, and depth at 432 × 432 with LLaVA-Video-7B.
  • Ablation Studies: Removing semantic similarity scoring reduces ScanQA from 29.8 to 27.1 EM@1 and Multi3DRefer from 56.2 to 52.2 F1@0.5.Removing modality-aware expert speculation also lowers SQA3D from 64.5 to 63.1 EM@1 and Multi3DRefer from 57.2 to 55.8 F1@0.5.
  • Ablation Studies: Adaptive modality selection achieves the best performance across all benchmarks, whereas adding more fixed modalities can introduce redundancy and interference.The analysis reports that RGB plus complementary 3D modalities often improves results, but larger fixed combinations do not consistently help.
  • Additional Analysis and Discussion: Training time falls to 47.4s per iteration versus 91.6s for Video-3D LLM and 125.2s for Ross3D, while inference latency remains comparable.The reported efficiency is attributed to adaptive modality filtering and sparse expert activation; SmartMage also remains competitive on RGB-only video benchmarks.

5 Conclusion

SmartMage concludes by presenting dynamic modality selection and modality-aware expert specialization as a unified approach to 3D scene understanding. Its evaluation supports strong 3D performance, competitiveness with RGB-only inputs, and question-dependent modality preferences.

  • 5 Conclusion: SmartMage decomposes multimodal reasoning into semantic-guided modality selection with SMART and modality-aware expert specialization with MAGE.The framework is presented as a unified and adaptive MLLM for 3D scene understanding.
  • 5 Conclusion: SmartMage achieves state-of-the-art performance on five standard 3D scene understanding benchmarks and outperforms prior methods under matched settings.The conclusion also reports competitive performance when only RGB inputs are available.
  • A.1 3D Scene Datasets: The unified corpus aggregates ScanNet-derived datasets covering question answering, situational reasoning, captioning, and grounding.The source datasets include ScanQA, SQA3D, Scan2Cap, ScanRefer, and Multi3DRefer.
  • Multimodal Dialogue Data and Prompting: Multimodal dialogue instances link conversations to 3D scenes and provide aligned RGB, BEV, depth, point cloud, and voxel inputs.Prompt templates support dense captioning, visual question answering, and visual grounding with modality placeholders.

A.3 Multi-Modal Preprocessing Pipeline

The preprocessing pipeline reduces redundant views, constructs complementary 3D representations, and produces stable multimodal inputs. FoVSR combines voxel pruning, coverage-based selection, and local sharpness refinement before BEV, depth, point, and voxel processing.

  • A.3.1 Keyframe Selection Using FoVSR: FoVSR selects a compact set of keyframes that preserves scene coverage while discarding redundant or low-quality views.This addresses the computational cost and redundancy of feeding tens to hundreds of RGB-D frames into the encoder.
  • A.3.1 Keyframe Selection Using FoVSR: FoVSR prunes uninformative voxels, computes per-view coverage, greedily selects views with maximum marginal coverage, and refines them using local sharpness.The final set balances spatial diversity with visual clarity.
  • A.3.1 Keyframe Selection Using FoVSR: FoVSR reduces selection complexity by using a pruned voxel set and computing each frame’s visible voxels once, unlike repeated full coverage evaluation.The comparison analyzes FoVSR against Maximum Coverage Sampling in both theoretical complexity and runtime.
  • A.3.2 3D-to-BEV Rendering: The BEV representation projects filtered reconstructed mesh geometry from an overhead orthographic camera onto the ground plane.Instance centers are added as object anchors, helping align object locations across modalities.
  • A.3.3 Depth Normalization and Completion: Depth preprocessing normalizes metric scale, clips invalid ranges, fills missing regions, and smooths neighboring keyframes for stable geometric cues.The completion step uses bilateral filtering and spatial consistency constraints.
  • A.3.4 Point and Voxel Construction: Point clouds sample 8,192 mesh points, while sparse voxels use 0.02 m cells, yielding complementary fine-grained and global spatial representations.Each point includes coordinates, color, surface normal, and semantic label; occupied voxels store occupancy and normalized color statistics.

B Model Architecture

SmartMage builds a unified representation from five complementary 3D modalities, then adapts both modality selection and expert processing to query semantics. Its SMART and MAGE modules provide global routing and modality-aware local specialization.

  • Omni-modal Feature Extraction: SmartMage aligns RGB, depth, BEV, point-cloud, and voxel features in a shared multimodal embedding space.The unified sequence combines appearance, metric depth, global layout, geometric surfaces, and volumetric context before routing.
  • 2D Modalities: RGB frames are selected as high-coverage keyframes, while depth embeddings add explicit spatial information through depth-derived 3D positional encoding.RGB features use FoVSR-selected multi-view images; depth values are back-projected using camera intrinsics before encoding.
  • 2D Modalities: BEV features encode global layout and object relationships, complementing egocentric RGB–depth observations.Orthographic projection produces a semantic map containing planar layout, room boundaries, object footprints, and instance masks.
  • 3D Modalities: PointNet++ encodes fine-grained point-cloud geometry, while Mask3D extracts hierarchical volumetric features from sparse voxels.The point branch uses coordinates, colors, normals, and semantic labels; the voxel branch produces embeddings after instance or segment aggregation.
  • Adaptive Routing: SMART schedules query-relevant modalities using semantic priors, text–modality similarity, and quality signals, while MAGE biases token routing toward modality-specialized experts.SMART performs global selection and token pruning; MAGE injects modality-aware affinity priors into sparse MoE gating.

C Task Setup

The evaluation spans three complementary 3D scene-understanding tasks: question answering, dense captioning, and visual grounding. Each task tests a different combination of scene comprehension, instance description, or object localization.

  • 3D Question Answering: 3D question answering evaluates holistic reasoning over objects, attributes, spatial relations, and commonsense.ScanQA uses spatial and attribute questions, while SQA3D adds sequential egocentric observations requiring viewpoint-change memory.
  • 3D Dense Captioning: 3D dense captioning detects scene objects and generates a free-form description for each instance.Scan2Cap descriptions incorporate category, appearance, material, and spatial relations; proposal embeddings condition caption generation.
  • 3D Visual Grounding: 3D visual grounding identifies the object set referred to by a natural-language expression.ScanRefer handles single-object localization, while Multi3DRefer extends grounding to multiple described instances and ambiguity resolution.
  • 3D Visual Grounding: Grounding encodes each proposal and referring expression jointly, then selects the proposal with the highest matching score.The selected proposal’s Mask3D bounding box becomes the final grounded 3D localization.
  • Evaluation Scope: Together, the tasks cover global comprehension, instance-level generation, and precise localization under diverse multimodal reasoning demands.This spectrum assesses reasoning, generation, and grounding across different input conditions and complexities.

D.1 ScanFacet Benchmark

ScanFacet is a diagnostic benchmark that reorganizes ScanQA and SQA3D questions into fine-grained semantic categories. It is designed to reveal which modality combinations support different kinds of 3D reasoning.

  • Benchmark Design: ScanFacet groups questions into eight semantic categories, including color, location, material, number, shape, object type, spatial relation, and other.The benchmark restructures question–answer pairs from ScanQA and SQA3D for semantic analysis.
  • Benchmark Design: An LLM-assisted pipeline performs semantic parsing, intent normalization, and self-consistency filtering before human review.Human annotators make minimal corrections, affecting less than 3% of samples.
  • Dataset Composition: The combined source datasets provide different category distributions, producing a semantically diverse benchmark for modality-preference analysis.ScanQA contributes more color, location, and object-type samples, while SQA3D contributes more number and spatial-relation questions.
  • Diagnostic Purpose: ScanFacet supports fine-grained comparisons between attribute-centric and relation-centric queries through a unified diagnostic interface.It is intended as a controlled diagnostic tool rather than another training dataset.

E.1 More Quantitative Comparisons

SmartMage reports strong results across question answering, dense captioning, and visual grounding, with adaptive routing performing especially well across semantic categories and grounding conditions. The benchmark analyses also examine robustness, ambiguity, and expert scaling.

  • ScanFacet Analysis: Across ScanFacet categories, semantic-adaptive routing achieves the highest F1 and CIDEr scores versus single-modality inputs and static fusion.RGB-centric combinations benefit color and material, while geometry-aware modalities benefit shape and spatial reasoning.
  • 3D Visual Grounding: 65.9% and 59.5% Acc@0.25/0.5 on ScanRefer and 65.4% and 60.7% F1@0.25/0.5 on Multi3DRefer establish state-of-the-art grounding results.The gains span single-reference and multi-reference settings involving subtle appearance cues, fine-grained geometry, and spatial relations.
  • 3D Visual Grounding: 56.7% Acc@0.25 and 53.5% Acc@0.5 on ambiguous ScanRefer samples show strong performance in the Multiple subset.SmartMage also reaches 89.6% and 84.5% on the Unique subset at the same thresholds.
  • 3D Dense Captioning: At IoU=0.25, SmartMage reaches BLEU-4 44.4, METEOR 30.9, and CIDEr 93.8 on Scan2Cap.At IoU=0.5, it reports BLEU-4 43.6 and METEOR 32.9.
  • Expert Analysis: Performance generally improves as the expert count increases from 4 to 20, although larger default settings were not tested because of GPU memory limitations.Deeper MoE placement also improves all reported ScanQA metrics.

E.3 Qualitative Results on ScanFacet

SmartMage handles diverse ScanFacet question types, including counting, color recognition, localization, spatial reasoning, and negative reasoning. Its routing analysis compares modality-specific expert trajectories with and without MES.

  • Qualitative Results: SmartMage handles counting, color recognition, object localization, spatial orientation, viewpoint-dependent queries, and absent-object reasoning on ScanFacet.Examples include counting chairs, identifying a brown chair, locating a radiator under a window, and recognizing when queried objects are absent.
  • Routing Visualization: Figure 15 compares top-10 modality routing trajectories across MoE layers with and without the MES module.Top-1 and Top-2 paths are highlighted for each modality, while remaining trajectories are shown in gray.
  • Routing Visualization: Without MES, expert routing is diffuse and inconsistent across layers, whereas MES produces clearer modality-aligned activation patterns.RGB tokens strongly activate E2 and E6, BEV tokens favor E3 and E7, and point-cloud tokens predominantly route to E5.
  • Routing Visualization: MES yields reduced routing dispersion and more consistent modality-specific expert preferences across representation depths.The resulting trajectories indicate more structured specialization of heterogeneous modalities toward suitable experts.

E.5 Failure Case Analysis

SmartMage’s failure cases involve annotation ambiguity and visual inconsistency, rather than a single uniform error source. The examples motivate attention to both ambiguous labels and robustness to multi-view appearance variation.

  • Failure Cases: An end-table grounding error reflects annotation ambiguity because both the predicted and ground-truth objects satisfy the description.The model selected object 13 instead of object 21, while both were circular and next to a teal couch.
  • Failure Cases: Inconsistent lighting and color across multi-view RGB images can impair recognition of subtle color differences.For the black-towel query, the model selected object 9 instead of the correct object 13.
  • Failure Cases: Object-frequency bias may also favor a frequently observed incorrect instance over a less prominent correct instance.The failure analysis suggests object 9’s higher occurrence across frames could bias attention away from object 13.

F Limitations and Broader Impacts

SmartMage’s broader scope is constrained by reduced and imperfect inputs, noisy training data, and risks in safety- or privacy-sensitive deployment. The authors therefore emphasize responsible development, transparent reporting, and human oversight.

  • Limitations: Key viewpoints may be missed by FoVSR keyframe selection, while FPS downsampling can remove fine-grained details from small structures.These reductions may degrade performance when other modalities provide insufficient complementary information.
  • Limitations: Blurry multi-view images and annotation inaccuracies can hinder precise spatial understanding and object localization.The limitation is tied to the quality of the training dataset and its visual and labeling noise.
  • Broader Impacts: Detailed spatial perception and multimodal reasoning could be misused in privacy-sensitive surveillance or military applications.The authors identify these as potential risks of stronger 3D scene understanding.
  • Broader Impacts: Inaccurate modality selection or spatial reasoning may lead to inappropriate actions when embodied predictions are deployed without sufficient verification.The authors also note that visual, geometric, and annotation errors can propagate to downstream decisions.
  • Broader Impacts: Safety- and privacy-sensitive deployments require responsible development, transparent limitation reporting, and careful human oversight.This recommendation follows the stated risks of misuse and downstream error propagation.
Loading 2608.05137v3…