PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision

Xiaofeng Han, Shunpeng Chen, Zenghuang Fu, Zhe Feng, Lue Fan, Dong An, Changwei Wang, Li Guo, Weiliang Meng, Xiaopeng Zhang, Rongtao Xu, Shibiao Xu

arXiv:2504.02477v3cs.ROcs.CV

TL;DR

Robot vision needs to overcome the perceptual limits of unimodal systems while handling heterogeneous data and demanding deployment constraints. This survey organizes multimodal fusion methods and VLMs across robotic tasks, compares their architectures and applicability, and reviews datasets and benchmarks. It identifies cross-modal alignment, efficient fusion, real-time deployment, and domain adaptation as central challenges, with lightweight and self-supervised approaches among the proposed directions.

  • Problem

    Unimodal perception is limited in complex environments, while multimodal robotic systems still face heterogeneous data integration, computational, and real-time deployment challenges.

  • Method

    The survey systematically reviews fusion paradigms and VLMs across robotic tasks, compares traditional and large-model approaches, and analyzes datasets and evaluation benchmarks.

  • Results

    The survey identifies cross-modal alignment, efficient training, real-time performance, and deployment as key research challenges across robot vision applications.

  • Takeaways & Limitations

    Future progress should emphasize lightweight fusion architectures, efficient pretraining, cross-modal self-supervised learning, and system optimization for real-world deployment.

  • Takeaways & Limitations

    Cloud-based computing frameworks often lack standardization for heterogeneous cloud-edge-device systems, limiting development efficiency and collaborative benefits in edge real-time fusion.

Abstract

from arXiv · show

Robot vision has greatly benefited from advancements in multimodal fusion techniques and vision-language models (VLMs). We adopt a task-oriented perspective to systematically review the applications and advancements of multimodal fusion methods and VLMs in the field of robot vision. For semantic scene understanding tasks, we categorize fusion approaches into encoder-decoder frameworks, attention-based architectures, and graph neural networks. Meanwhile, we also analyze the architectural characteristics and practical implementations of these fusion strategies in key tasks such as simultaneous localization and mapping (SLAM), 3D object detection, navigation, and manipulation. We compare the evolutionary paths and applicability of VLMs based on large language models (LLMs) with traditional multimodal fusion methods.Additionally, we conduct an in-depth analysis of commonly used datasets, evaluating their applicability and challenges in real-world robotic scenarios. Building on this analysis, we identify key challenges in current research, including cross-modal alignment, efficient fusion, real-time deployment, and domain adaptation. We propose future directions such as self-supervised learning for robust multimodal representations, structured spatial memory and environment modeling to enhance spatial intelligence, and the integration of adversarial robustness and human feedback mechanisms to enable ethically aligned system deployment. Through a comprehensive review, comparative analysis, and forward-looking discussion, we provide a valuable reference for advancing multimodal perception and interaction in robotic vision. A comprehensive list of studies in this survey is available at https://github.com/Xiaofeng-Han-Res/MF-RV.

1. Introduction

This survey reviews multimodal fusion and vision-language models for robot vision, emphasizing their applications across core robotic tasks and their complementary capabilities. It also analyzes datasets, identifies deployment challenges, and proposes directions for more intelligent and generalizable systems.

  • Motivation: Multimodal fusion combines complementary sensory information to improve perception, reasoning, and decision-making beyond unimodal approaches in complex environments.The survey discusses inputs including vision, language, depth, LiDAR, and tactile data.
  • Scope: The survey covers semantic scene understanding, SLAM, 3D object detection, embodied navigation, robotic manipulation, and visual localization.Its task-oriented scope extends beyond basic segmentation and detection to complex robotic applications.
  • Comparative analysis: It systematically compares traditional multimodal fusion methods with vision-language models in architecture, function, applicable tasks, and integration potential.The comparison addresses their connections and complementary strengths.
  • System capabilities: Multimodal systems are presented as offering greater perceptual robustness, semantic expressiveness, cross-modal alignment, and high-level reasoning in dynamic or partially observable environments.These advantages are framed as practical value relative to unimodal approaches.
  • Datasets: The survey analyzes robotics datasets by modal combinations, covered tasks, applicable scenarios, and limitations to support benchmarking and model evaluation.This analysis highlights dataset applicability and challenges in robotic scenarios.
  • Challenges and future directions: Key challenges include cross-modal alignment, efficient training, real-time optimization, and deployment, motivating lightweight architectures and cross-modal self-supervised learning.The proposed directions target more intelligent, generalizable, and deployable robotic vision systems.

2.1. Semantic Scene Understanding

Multimodal fusion improves semantic scene understanding by combining complementary sensor information and is organized around encoder-decoder, attention-based, and graph-based strategies. These approaches support richer scene representations, but attention-based fusion faces substantial computational costs for long or high-resolution inputs.

  • Multimodal fusion combines vision, language, depth, LiDAR, and speech to reduce sensor limitations and improve robust scene interpretation.
  • Encoder-Decoder Framework: Encoder-decoder frameworks independently encode modalities and use decoders to fuse their representations into task outputs.Cross-modal fusion commonly occurs through early or deep fusion, often enhanced with attention mechanisms.
  • Attention Mechanism: Attention mechanisms adaptively weight cross-modal features, capture long-range dependencies, and improve multimodal alignment and representation.Unified architectures also simplify preprocessing and multimodal integration.
  • Attention Mechanism: Attention-based fusion supports 3D scene completion, segmentation, and object detection, but some implementations rely on offline processing that limits real-time applicability.Multi-Sem Fusion combines LiDAR and camera data with adaptive attention, while CLFT uses projection and cross-fusion in a Transformer decoder.
  • Attention Mechanism: Standard self-attention has quadratic time and memory complexity in input length, creating bottlenecks for long sequences and high-resolution multimodal inputs.Sparse, linear, and low-rank attention reduce this burden but introduce a trade-off between efficiency and alignment accuracy.
  • Graph Neural Network: GNN-based fusion maps multimodal data to graph nodes and edges, applies message passing and feature aggregation, and integrates modality-specific GNN outputs for reasoning.This design models structured relationships within modalities before final decision fusion.

2.2. 3D Object Detection

Multimodal 3D object detection combines complementary sensor information through choices about fusion timing, inputs, granularity, and attention. The survey traces progression from early LiDAR-camera fusion toward adaptive and attention-based methods while noting computational and environmental limitations.

  • Fusion Strategies: Fusion design is organized by when to fuse, what to fuse, and how to fuse multimodal data.Timing includes early, intermediate, and late fusion; inputs include image and LiDAR representations; granularity includes ROI-wise, voxel-wise, and point-wise fusion.
  • Lidar+Camera: LiDAR-camera fusion combines LiDAR depth and spatial structure with camera-derived semantic and visual information.Single-camera detection lacks direct depth and is vulnerable to occlusion and adverse weather, while LiDAR is sparse at long distances and lacks color and texture.
  • Lidar+Camera: MV3D pioneered multi-view LiDAR-camera fusion, followed by methods exploring ROI-wise, voxel-wise, and point-wise feature integration.MV3D uses bird’s-eye-view, front-view, and RGB features; later methods include AVOD, PointFusion, and MVX-Net.
  • Attention-Based Fusion: Attention-based methods dynamically weight multimodal features and improve alignment, but their added complexity can limit embedded deployment.Examples include adaptive gating, cross-attention, soft association, and temporal fusion mechanisms.
  • Limitations: Instance-guided and multi-level fusion can struggle in dynamic, cluttered, occluded, or partially observable robotic scenes and impose substantial computational overhead.The survey identifies dependence on high-quality instance features and lower inference efficiency as practical limitations.
  • Other Sensor Fusion: Radar-camera and radar-LiDAR fusion add velocity, depth, or adverse-weather robustness to multimodal 3D detection.Representative approaches include RRPN, CenterFusion, and MVDNet, while 4D radar contributes velocity measurements for dynamic-object tracking.

2.3. Navigation and Localization

Robot navigation and localization use multimodal perception, language, spatial representations, and interaction to support goal-directed behavior in complex environments. The survey highlights instruction following, dialogue, topological maps, and multimodal localization as complementary directions, while noting deployment limitations.

  • Embodied Navigation: Embodied navigation uses visual, tactile, and auditory information across goal-oriented, instruction-following, and dialogue-based settings.These settings differ in whether agents receive goals, natural-language commands, or interactive guidance during navigation.
  • Goal-Oriented Navigation: Goal-oriented navigation combines visual semantics and spatial priors to explore environments and reach target locations without external instructions.The survey cites Habitat ObjectNav as a validation setting for this capability.
  • Instruction-Following Navigation: Instruction-following methods can generate executable trajectories from language commands and support zero-shot cross-task planning.InstructNav uses Dynamic Chain-of-Navigation and Multi-sourced Value Maps, while NaVid uses a video-driven VLM.
  • Topological Navigation: Topological maps integrate current observations with existing environment models to expand navigable regions and support globally informed path planning.They can also help mitigate disorientation from random exploration and support recovery after navigation errors.
  • Limitations: Navigation systems remain constrained by degraded visual-semantic alignment in low light, occlusion, and sensory noise, plus long reasoning pipelines on embedded platforms.Complex prompt designs and resource constraints create efficiency bottlenecks during deployment.
  • Dialogue-Based Navigation: Dialogue-based navigation lets agents clarify ambiguous instructions, obtain environmental information, and correct navigation errors during execution.This enables dynamic path adjustment rather than strict adherence to predefined instructions.
  • Navigation Outcomes: Multimodal fusion improves navigation flexibility by combining vision, language, and spatial cues for visual-goal and language-guided tasks.The survey characterizes these combinations as supporting more robust navigation in diverse and dynamic environments.
  • Visual Localization: Multimodal localization combines RGB, depth, and inertial measurements, while hybrid geometric-neural systems address scale ambiguity and drift.D3VO uses uncertainty estimation, and learned depth and optical flow support consistent pose estimation.

2.4. Simultaneous Localization and Mapping

Multimodal SLAM addresses the limitations of single-sensor systems by combining complementary geometric, semantic, inertial, and localization information. The survey describes an evolution from loosely coupled methods toward tightly integrated optimization and attention-based frameworks, with future emphasis on semantic understanding and efficient deployment.

  • Motivation: Single-modal visual and LiDAR SLAM are limited by their sensor properties, especially in low-light, textureless, sparse, or low-reflectivity environments.Visual SLAM struggles with weak visual cues, while LiDAR lacks semantic understanding and can become sparse at long range.
  • Multimodal Fusion: Multimodal SLAM combines cameras, LiDAR, IMUs, GPS, and radar to improve environmental perception and motion estimation.LiDAR contributes spatial structure, vision contributes semantic detail, and IMU compensates for short-term motion-estimation uncertainty.
  • Evolution of SLAM: Early systems such as V-LOAM and LIMO combined visual, LiDAR, and inertial information to exploit complementary geometric, texture, and motion cues.Later systems including LIC-Fusion and LIC-Fusion 2.0 adopted tightly coupled optimization frameworks.
  • Attention and Transformers: Transformer-based methods model temporal and cross-modal relationships, with radar integration adding robustness under adverse weather conditions.UVIO jointly learns from vision and IMU, while RD-VIO incorporates radar for tracking and estimation.
  • Representative Systems: Factor-graph and vision-IMU-GPS systems support multimodal localization in low-texture, dynamic, large-scale, and high-precision settings.LVI-SAM tightly couples visual and LiDAR data, while VINS-Fusion is used in UAVs and mobile robots.
  • Summary: Multimodal SLAM has progressed from loosely coupled architectures to deeply integrated frameworks with stronger cross-modal alignment and environmental understanding.The survey connects this progression to semantic SLAM and future foundation-model, self-supervised, and lightweight real-time directions.

2.5. Robot Manipulation

Robot manipulation increasingly uses multimodal fusion to connect perception, language, touch, and action planning. Vision-language-action models support semantic task understanding and control, while vision-touch fusion improves grasping precision, stability, and force adjustment.

  • Vision-Language-Action Models: Vision-Language-Action models unify visual perception, language understanding, and action planning for complex manipulation tasks.Their central design is a shared representation that supports precise action planning and execution.
  • Vision-Language-Action Models: Language integration extends visual manipulation systems by connecting natural-language instructions to robot control commands.RT-2 aligns visual and language representations through pretraining and generates control commands from linguistic input.
  • Manipulation Outcomes: VLA-based fusion strengthens task-semantic understanding, action planning, and decision-making in dynamic and complex environments.The survey presents these methods as combining complementary visual, language, and action modalities.
  • Vision-Touch Fusion: Vision-touch fusion combines global visual geometry with local tactile force, surface, and slip information for robotic grasping.Vision supports position and orientation estimation, while touch contributes surface details and force feedback for grasp-point selection.
  • Vision-Touch Fusion: Vision-touch methods support grasp generation, planning, stability assessment, and dynamic force adjustment while compensating for single-modality limitations.The survey identifies improved grasp-path precision and adaptive grasping forces as key capabilities.

3. Evolution of Vision-Language Models

This section traces vision-language models from multimodal pre-training and alignment methods to Transformer-based architectures and instruction tuning. It also highlights deployment constraints, including the need for cross-platform frameworks in real-time edge fusion.

  • 3.1. Pre-Training Across Modalities: Pre-training improves vision-language performance, enables knowledge transfer across modalities, reduces reliance on labeled data, and strengthens generalization.
  • 3.1. Pre-Training Across Modalities: Cross-modal pre-training jointly models modalities in a shared embedding space to learn associations and complementarities between them.This approach supports cross-modal understanding and alignment across inputs such as images, text, audio, and video.
  • 3.2. Cross-Modal Alignment and Representation Learning: Representation learning uses contrastive, self-supervised, and cross-modal generative methods to improve multimodal alignment and interaction.Self-supervised learning can use unlabeled data, while generative models produce one modality from another.
  • 3.3. Transformer Variants and Large Vision-Language Models: Transformers became central to multimodal learning because self-attention models long-range dependencies while parallel computation avoids recurrent sequential constraints.Efficient variants reduce resource use through sparse attention, low-rank projections, or locality-sensitive hashing.
  • 3.4. Multimodal Instruction Following and Learning: Instruction tuning combines supervised fine-tuning with human-feedback methods to improve task execution, controllability, reliability, and alignment with user intent.Multimodal tuning can jointly optimize vision encoders and text decoders; RLHF uses preference data, reward modeling, and reinforcement learning.
  • 3.6. Lightweight and Real-Time Models: Real-time edge fusion remains constrained by cloud-oriented frameworks that do not adequately support heterogeneous cloud-edge-device systems.The section identifies a need for universal, cross-platform computational frameworks.

4. Datasets and Benchmarks for Multimodal Fusion

This section reviews multimodal datasets spanning semantic understanding, autonomous driving, indoor scenes, manipulation, grasping, tactile perception, and navigation. Across these benchmarks, modality diversity supports training and evaluation, while limited diversity, coarse annotations, and inconsistent standards remain important gaps.

  • Dataset Scope: Multimodal robotic datasets combine modalities such as RGB, depth, LiDAR, radar, audio, text, and tactile sensing across indoor and outdoor tasks.They support scene understanding, navigation, manipulation, and related multimodal fusion research.
  • Semantic Scene Understanding: Indoor benchmarks including Matterport3D, ScanNet, NYU Depth V2, and Hypersim support semantic segmentation, 3D reconstruction, and multi-task learning.Matterport3D provides 10,800 panoramic RGB-D views, while ScanNet contains 2.5 million RGB-D frames across 1,513 scenes.
  • Cross-Modal and Interactive Scenarios: Interactive datasets such as 360+x and ScanQA connect multimodal perception with multi-view understanding, language-guided object localization, and 3D scene reasoning.
  • Manipulation and Grasping: Robotic manipulation datasets cover multi-task control, grasp detection, language-conditioned grasping, and tactile-visual perception.Examples include DROID, RT-1, R2SGrasp, Grasp-Anything, Touch and Go, and VisGel.
  • Dataset Limitations: Dataset limitations include coarse task annotations, restricted environments and object diversity, insufficient real-world complexity, and inconsistent evaluation standards.The survey calls for unified benchmarks with broader scenarios, fine-grained annotations, and cross-task generalization assessments.

5. Performance Evaluation of Fusion Techniques

The survey evaluates multimodal fusion and VLM methods across 3D detection, navigation, and 3D question answering using task-specific metrics and benchmarks. Results indicate that modality-aware fusion, structured memory, environment adaptation, and hierarchical cross-modal alignment improve robustness and generalization in challenging robotic scenarios.

  • Evaluation covers 3D object detection, semantic scene understanding, 3D visual question answering, and robotic manipulation using task-specific benchmarks and metrics.The comparison draws primarily on original or aligned reproduced results.
  • 3D Object Detection: SpaRC improves small-object AP through spatial-semantic consistency, while RQR3D reaches 59.7% mAP with balanced localization of small and medium-sized targets.SpaRC addresses semantic drift through policy-based cross-modal alignment; RQR3D uses recursive query refinement for image features.
  • 3D Object Detection: LiDAR-camera fusion provides high-precision perception, while radar-camera fusion is more adaptable in challenging environments; Transformer architectures improve heterogeneous information integration.LiDAR contributes geometric information, cameras contribute semantics, and radar contributes motion information and adverse-condition robustness.
  • Navigation: Navigation methods increasingly combine structured memory, environment augmentation, multimodal pretraining, and transformer-based planning to improve efficiency and generalization in unseen environments.The surveyed progression moves from sequence modeling toward semantic understanding-driven navigation.
  • Navigation: PanoGen++ achieves 75% SR and 64% SPL, while VISTA achieves 75% SR and 67% SPL on the Room-to-Room test-unseen setting.PanoGen++ uses panoramic semantic graph generation, whereas VISTA combines curriculum training with multimodal language-vision alignment.
  • 3D Question Answering and Scene Understanding: BridgeQA achieves state-of-the-art ScanQA performance by linking global 3D scene context with fine-grained object features through hierarchical cross-modal bridging.The approach improves spatial reasoning and robustness for complex spatial relationships and ambiguous semantic queries.

6. Challenges and Opportunities

Multimodal fusion and VLMs face challenges spanning data quality, modality heterogeneity, computational efficiency, dataset scarcity, environmental bias, spatial understanding, and ethical deployment. The survey highlights corresponding directions including denoising, adaptive fusion, efficient architectures, synthetic and self-supervised data, spatial memory, and adversarial evaluation.

  • 6.1. Low-quality Data: Low-quality data introduce noise, missing modalities, quality imbalance, and dynamic changes that undermine multimodal model reliability.Noise may be modality-specific or cross-modal, while missing modalities arise from device issues, data loss, or user preference.
  • 6.2. Heterogeneity: Heterogeneous modality structures and feature distributions make unified representation learning, alignment, and information interaction central fusion problems.Images, audio, and text differ in dimensionality, continuity, temporal structure, and symbolic representation.
  • 6.3. Efficient Training and Inference: Large-scale VLMs create training and deployment bottlenecks through billions of parameters, dense attention, long contexts, latency, memory use, and energy costs.Dynamic modality selection, knowledge distillation, lightweight architectures, and modality dropout are identified as efficiency directions.
  • 6.4. Dataset Limitations and Environmental Bias: Visual-language datasets remain limited by annotation noise, insufficient semantic diversity, expensive annotation, and environmental bias toward favorable conditions.Synthetic data, self-supervised learning, simulation, and domain-specific physical-world datasets are proposed to address these constraints.
  • 6.5. Perspectives and future directions: Current multimodal models still lack explicit spatial topology and persistent spatial representations needed for reasoning across space, language, and action.Suggested directions include structured spatial memory, 3D graphs, topological embeddings, SLAM, egocentric modeling, and loop closure.
  • 6.5. Perspectives and future directions: VLM deployment in robots raises safety and accountability concerns because image perturbations and ambiguous language can produce incorrect interpretations and unsafe actions.The survey calls for adversarial benchmarks, causal reasoning, multimodal consistency regularization, and traceable action justifications.

7. Conclusions

The survey synthesizes multimodal fusion and VLM research across robot-vision tasks, modalities, datasets, and evaluation benchmarks. It identifies cross-modal alignment, lightweight adaptation, and task-oriented online fusion as central directions for deploying capable models on robots.

  • 7. Conclusions: The survey compares encoder-decoder, attention-based, and graph-based fusion paradigms across semantic understanding, 3D detection, navigation, manipulation, and related tasks.It also examines modalities including vision, language, depth, and point clouds, alongside datasets and evaluation benchmarks.
  • 7. Conclusions: Cross-modal alignment is critical because modal differences and mismatched semantic granularity constrain robot perception performance.The conclusion identifies alignment strategies as a determinant of the perception system’s performance ceiling.
  • 7. Conclusions: Lightweight, multi-stage adaptation is presented as practically necessary for deploying large-scale VLMs on resource-limited robot platforms.The conclusion emphasizes edge-computing constraints as a key deployment consideration.
  • 7. Conclusions: Task-oriented and online adaptation fusion strategies are becoming an increasingly important research focus for robot vision.
Loading 2504.02477v3…