PaperlayerView exact revision on arXiv

Source-linked AI summary

Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems

Song Wang, Lingdong Kong, Xiaolu Liu, Hao Shi, Wentong Li, Jianke Zhu, Steven C. H. Hoi

arXiv:2512.24385v2cs.CVcs.RO

TL;DR

Autonomous systems need multi-modal representations that integrate complementary sensor data, while annotation-intensive learning and cross-sensor integration remain challenges. This paper surveys datasets, pre-training paradigms, and open-world applications; its analysis characterizes camera-LiDAR integration as producing semantically rich and geometrically precise representations.

  • Problem

    Annotation-heavy supervised learning creates scalability and generalization bottlenecks, while integrating heterogeneous sensors into holistic perception remains a challenge.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically analyzes multi-modal pre-training, platform-specific datasets, sensor interactions, and applications across autonomous systems.

    Source: §1 · p. 3

  • Results

    The analysis concludes that integrating camera and LiDAR produces representations that are both semantically rich and geometrically precise.

    Source: §7 · p. 21

  • Takeaways & Limitations

    The paper characterizes unified foundation models as central to spatial intelligence and calls for future work on world models, VLA frameworks, and explicit reasoning.

    Source: §7 · p. 21

    Distilling large teacher models into lightweight, real-time student networks without catastrophic performance drops remains a deployment bottleneck under onboard latency and power constraints.

    Source: §6.1 · p. 21

Abstract

from arXiv · show

The rapid advancement of autonomous systems, including self-driving vehicles and drones, has intensified the need to forge true Spatial Intelligence from multi-modal onboard sensor data. While foundation models excel in single-modal contexts, integrating their capabilities across diverse sensors like cameras and LiDAR to create a unified understanding remains a formidable challenge. This paper presents a comprehensive framework for multi-modal pre-training, identifying the core set of techniques driving progress toward this goal. We dissect the interplay between foundational sensor characteristics and learning strategies, evaluating the role of platform-specific datasets in enabling these advancements. Our central contribution is the formulation of a unified taxonomy for pre-training paradigms: ranging from single-modality baselines to sophisticated unified frameworks that learn holistic representations for advanced tasks like 3D object detection and semantic occupancy prediction. Furthermore, we investigate the integration of textual inputs and occupancy representations to facilitate open-world perception and planning. Finally, we identify critical bottlenecks, such as computational efficiency and model scalability, and propose a roadmap toward general-purpose multi-modal foundation models capable of achieving robust Spatial Intelligence for real-world deployment.

1 Introduction

Autonomous systems need holistic perception from complementary onboard sensors, but annotation-heavy learning constrains scale and generalization. This paper surveys multi-modal pre-training, datasets, and applications, connecting modality-specific methods to unified frameworks.

1 Introduction

  • Camera semantics, LiDAR geometry, radar motion cues, and event-camera timing are complementary inputs to holistic, safe, and generalizable autonomy.The paper frames effective integration of these heterogeneous streams as central to perception.Source: §1 · p. 1
  • Annotation-heavy supervised learning creates scalability and generalization bottlenecks, motivating representation learning and foundation models trained from raw sensor data.Foundation models are presented as a route to transferable representations across domains.Source: §1 · p. 2
  • The paper analyzes single-modality, cross-modal, and unified pre-training strategies in relation to sensor characteristics, platform constraints, and downstream applications.It also examines data sparsity, sensor noise, multi-modal alignment, and real-time processing demands.Source: §1 · pp. 2–3

1.2 Relation to Previous Studies

  • Earlier surveys often focus on individual modalities, platforms, or tasks; this work connects those areas through a framework centered on foundation models and multi-modal representation learning.Its scope primarily covers onboard camera and LiDAR, while also examining representative radar and event-camera methods.Source: §1.2 · p. 3, §1.1 · p. 3

1.3 Organization

  • The paper proceeds from sensor and learning foundations through platform datasets and pre-training methods to open-world applications, challenges, and future directions.The roadmap concludes with a summary of its main insights.Source: §1.3 · pp. 3–4

2 Background

Spatial intelligence depends on learning from heterogeneous onboard sensors whose semantic, geometric, and temporal strengths differ. The section traces learning paradigms from self-supervision and cross-modal interaction to knowledge transfer and foundation models.

2 Background

  • Multi-modal pre-training aims to distill compact, discriminative, semantically rich representations that support open-world generalization and embodied reasoning.The paper frames this as a foundation for spatial intelligence in autonomous systems.Source: §2 · p. 4

2.1 Onboard Sensors and Data Characteristics

  • Cameras offer dense semantics but are weather- and lighting-sensitive, while LiDAR provides precise yet sparse geometry, radar velocity cues, and event cameras fine-grained timing.Representation learning must address these sensor disparities to build a coherent world model.Source: §2.1 · p. 4

2.2 Paradigms of Representation Learning

  • The learning trajectory progresses from self-supervision on unlabeled data to cross-modality interaction, vision-model knowledge transfer, and unified foundation or generative world models.These paradigms are presented as a progression toward holistic representations.Source: §2.2 · p. 4

2.2.1 Self-Supervised Learning

  • Self-supervised learning uses signals inherent in unlabeled data, including contrastive views, masked reconstruction, and generative prediction of future frames or occupancy.Future prediction is described as a precursor to world models because it can capture environmental physics and dynamics.Source: §2.2.1 · p. 4

2.2.2 Cross-Modality Interaction

  • Cross-modal methods fuse sensor streams into unified representations, for example by projecting camera features onto LiDAR points, while addressing alignment, noise, and missing modalities.The stated goal is greater robustness and semantic depth.Source: §2.2.2 · pp. 4–5

2.2.3 Knowledge Distillation and Transfer Learning

  • Knowledge distillation transfers vision foundation models’ open-vocabulary capabilities and semantics to 3D sensor backbones without requiring massive annotated 3D datasets.The passage describes 2D vision models as teachers and 3D backbones as students, bridging the data-scale gap between image and robotics domains.Source: §2.2.3 · p. 5

2.2.4 Foundation Models

  • Foundation models extend visual priors to LiDAR and radar through cross-modal alignment, while generative world models and VLA systems target future simulation and action planning.VLA models integrate vision, language, and action into a unified reasoning framework.Source: §2.2.4 · p. 5

3 Platform-specific Datasets

Platform-specific datasets differ in sensor coverage, synchronization, annotations, and domain diversity, shaping the learning strategies available to autonomous systems. The survey covers vehicles, drones, and other robots, and highlights datasets’ growing role in reasoning, simulation, and world modeling.

3.1 Overview of Sensor Modalities and Datasets

  • Dataset suitability for pre-training depends on sensor configuration, synchronization, annotation granularity, and diversity across weather, lighting, and geography.The section evaluates datasets across vehicles, drones, and other robotic platforms for tasks including open-vocabulary perception and generative world modeling.Source: §3.1 · p. 7, §3 · p. 7

3.2.1 Autonomous Vehicles

  • Vehicle datasets pair synchronized camera, LiDAR, and radar streams with self-supervised tasks such as future prediction, masked reconstruction, and contrastive distillation.Newer benchmarks add maps, long-horizon trajectories, and language-based scenarios alongside perception annotations.Source: §3.2.1 · p. 8

3.2.2 Drones (UAVs)

  • UAV datasets address bird’s-eye viewpoints, 6-DoF motion, perspective distortion, and blur through transfer learning, cross-view localization, and self-supervised flow estimation.Emerging multi-modal datasets extend the domain from 2D ground planes to 3D volumetric spaces.Source: §3.2.2 · p. 8, Table 2 · p. 8

3.2.3 Other Robotic Platforms

  • Other robotic datasets address platform-specific noise and motion, including water reflections and legged-robot camera shake, with simulation-to-real transfer and synthetic data supplementing scarce samples.The surveyed domains include railways, surface vehicles, and legged robots.Source: §3.2.3 · p. 9, Table 3 · p. 9

3.3 Key Dataset Trends and Implications

  • Datasets increasingly support reasoning and action through planning trajectories, logic-based scenarios, and open-vocabulary labels, while synthetic environments address real-world long-tail limits.The paper also notes data-volume and modality growth as a reason for self-supervised pre-training and automated data curation, labeling, and alignment.Source: §3.3 · p. 9
  • Datasets are shifting from static evaluation benchmarks toward active components in training reasoning-capable agents and generative world models.The paper connects rich multi-modal contexts with a transition from passive perception to world modeling and decision-making.Source: §3.3 · pp. 9–10

4 Pre-Training Techniques for Perception

The section organizes pre-training by sensor interaction and also examines how foundation-model priors and generative objectives extend methods toward geometric reasoning and downstream perception.

4 Pre-Training Techniques for Perception

  • Pre-training methods are grouped into single-modality baselines, camera- or LiDAR-centric cross-modal distillation, and unified frameworks jointly optimizing cross-modal encoders.The taxonomy organizes the methods by sensor interaction.Source: §4 · p. 10
  • Recent approaches use generative reconstruction and future forecasting alongside foundation-model semantic priors, with radar and event cameras also covered.The section synthesizes benchmark performance to assess how pre-training translates to downstream perception tasks.Source: §4 · p. 10

4.1 Single-Modality Pre-Training

Single-modality pre-training learns intrinsic semantic and geometric features from individual sensor streams without cross-modal supervision, establishing a foundation for sensor fusion. LiDAR methods emphasize geometry and temporal prediction, while camera methods progress toward 3D geometry and dynamics.

4.1 Single-Modality Pre-Training

  • Single-modality pre-training extracts intrinsic semantic and geometric features from individual sensor streams without cross-modal supervision.The section presents these representations as a prerequisite for effective sensor fusion.Source: §4.1 · p. 10

4.1.1 LiDAR-Only Pre-Training

  • LiDAR-only methods use masked reconstruction for structure, contrastive learning for spatial invariance, and forecasting for dynamic scene prediction.Methods include voxel or grid reconstruction, hierarchical contrastive objectives, and prediction of occupancy, flow, or point-cloud sequences.Source: §4.1.1 · p. 11

4.1.2 Camera-Only Pre-Training

  • Camera-only pre-training progresses from domain and temporal consistency to BEV lifting and implicit 3D representations, supporting geometry and dynamics learning.Examples include domain-affiliation and feature-ordering tasks, label-efficient BEV mapping, masked volumetric reconstruction, and voxel warping with multi-frame consistency.Source: §4.1.2 · p. 12

4.2 Multi-Modality Pre-Training

Multi-modality pre-training bridges semantic and geometric information through LiDAR-centric transfer, camera-centric geometric supervision, and unified frameworks. These approaches progress from asymmetric knowledge transfer toward jointly learned representations.

4.2 Multi-Modality Pre-Training

  • The taxonomy distinguishes LiDAR-centric transfer of visual semantics, camera-centric injection of geometric priors, and unified modality-agnostic representation learning.The approaches are categorized by the direction of information flow between modalities.Source: §4.2 · p. 12

4.2.1 LiDAR-Centric Pre-Training

  • LiDAR-centric methods transfer image semantics into 3D through guided reconstruction, pixel-point alignment, foundation-model distillation, and temporal motion transfer.These approaches target semantic awareness in point-cloud networks, with reported gains in detection and segmentation, particularly in data-scarce regimes.Source: §4.2.1 · pp. 12–13

4.2.2 Camera-Centric Pre-Training

  • Camera-centric methods use LiDAR supervision for depth, occupancy, and feature learning, then extend to forecasting and rendering for dynamic world modeling.Forecasting approaches predict future points or 4D occupancy, while rendering-based methods use differentiable neural fields and 3D Gaussian Splatting.Source: §4.2.2 · pp. 13–14

4.2.3 Unified Pre-Training

  • Unified frameworks mask and encode camera and LiDAR inputs, transform features into a common coordinate system, fuse them, and reconstruct sensor information.The shared representation is typically BEV or volumetric, and holistic reconstruction targets geometry, image texture, and sometimes depth maps.Source: §4.2.3 · p. 15

4.3 Incorporating Additional Sensors

Additional sensors complement camera and LiDAR for robustness under challenging conditions. The section surveys radar and event-camera pre-training, alongside auxiliary signals used as supervision or constraints.

4.3 Incorporating Additional Sensors

  • Radar and event cameras can complement camera-LiDAR perception under adverse weather, high-speed motion, and extreme lighting conditions.Radar offers resilience in fog and rain, while event cameras provide high temporal resolution and high dynamic range.Source: §4.3 · p. 15

4.3.1 Radar Pre-Training

  • Radar pre-training aligns sparse returns with richer modalities, reconstructs masked signals to learn priors and suppress noise, and adapts across domains.Radar-specific masking and curriculum learning address high signal-quality variance; lightweight inductive biases are explored for edge devices.Source: §4.3.1 · p. 16

4.3.2 Event Camera Pre-Training

  • Event-camera methods reconstruct spatial structure or predict future frames, while cross-modal alignment with RGB supports semantic understanding in motion blur.Specialized transformers model event sparsity and temporal continuity; event-RGB synergy is also applied to agile aerial navigation.Source: §4.3.2 · p. 16

4.3.3 Auxiliary Modalities

  • IMUs, GPS, and thermal sensors provide auxiliary constraints or supervision through ego-motion consistency, scene retrieval or trajectory consistency, and RGB-to-thermal adaptation.Thermal data can provide signatures for living beings that are invisible to standard cameras at night.Source: §4.3.3 · p. 16

4.4 Empirical Analysis and Benchmark Performance

Benchmarks report gains from multi-modal detection pre-training and visual-prior transfer for LiDAR segmentation. The cited results include stronger performance in data-scarce regimes and a full-dataset result for LiMoE.

4.4.1 3D Object Detection

  • 71.1 mAP and 73.8 NDS for UniM2AE exceed the reported FocalFormer3D baseline comparison, while UniPAD gains +4.5 mAP over UVTR-M to reach 69.9 mAP.The passage attributes the pattern to jointly optimized multi-modal encoders outperforming camera-only baselines.Source: §4.4.1 · p. 17

4.4.2 LiDAR Segmentation

  • 77.27 mIoU for LiMoE on the full dataset accompanies the reported finding that 3D backbones can inherit open-world semantics from ViT-S teachers.The section describes this transfer as a Scaling Law Transfer phenomenon.Source: §4.4.2 · pp. 17–18
  • 30.30 mIoU for the random baseline at 1% labeled data compares with 50.58 mIoU for OLIVINE and 49.60 mIoU for LiMoE.The reported advantage of visual-prior approaches is magnified in data-scarce regimes.Source: §4.4.2 · p. 17, Table 7 · p. 18

5 Open-World Perception and Planning

The section traces a path from open-world perception challenges through text-grounded supervision to world models that simulate futures and support action. It contrasts generative planning with latent and language-based action architectures.

5.1 Open-World Challenges

  • Open-world deployment requires novel-object recognition, robustness to domain shifts and anomalies, and learning from diverse unlabeled data because manual annotation cannot cover corner cases.The section also calls for quantifying epistemic uncertainty to handle unknown unknowns safely.Source: §5.1 · p. 18

5.2 Text-Grounded Understanding

  • Text-aligned 2D features provide continuous supervision for self-supervised 3D occupancy prediction, whose approaches rival supervised baselines.The section also identifies 3D Gaussian Splatting as a direction toward semantically rich, geometrically continuous, renderable representations.Source: §5.2 · p. 19

5.3 Unified World Representation for Action

  • Generative planners predict future 3D-world evolution conditioned on ego-actions and significantly outperform discriminative baselines in collision rates and open-loop planning metrics.OccWorld and GenAD use objectives such as 4D occupancy flow; Table 9 evaluates planning L2 error and collision rate, with lower values indicating better safety and precision.Source: §5.3 · p. 19, Table 9 · p. 20
  • Vision-Action latent models learn future rewards and control signals from sensor inputs without perception labels or human annotations.They abstract the environment into high-dimensional latent states.Source: §5.3 · pp. 19–20
  • Vision-Language-Action frameworks place visual tokens and text in an LLM context to generate actions while supporting causal reasoning and complex social interactions.The section describes this as a unified autoregressive process.Source: §5.3 · p. 20

6 Challenges and Future Directions

The paper identifies remaining challenges for robust autonomy and outlines future directions spanning physically consistent world models, real-time embodied VLA, continuous semantic-geometric representations, and long-tail reasoning.

6.1 Current Challenges

  • VLMs excel at open-vocabulary recognition but often lack the fine-grained spatial grounding needed for centimeter-level localization, leaving semantic-geometric alignment a major challenge.The paper characterizes bridging semantic reasoning and low-level geometric constraints without compromising either as a theoretical and engineering challenge.Source: §6.1 · p. 20
  • Pre-training objectives treat samples equally, wasting computation on repetitive patterns while under-weighting rare, high-value events; auto-labeling also introduces hard-to-filter epistemic uncertainty.The paper frames the data challenge as mining long-tail corner cases and safety-critical scenarios rather than simply acquiring more data.Source: §6.1 · pp. 20–21
  • Distilling large foundation-model teachers into lightweight real-time students without catastrophic performance drops remains a deployment bottleneck under onboard latency and power constraints.The paper contrasts constrained edge deployment with cloud-based pre-training resources.Source: §6.1 · p. 21

6.2 Future Directions

  • Future world models should enforce physical consistency with differentiable physics or geometric constraints so their simulated interactions can serve as reliable training environments for safety-critical policies.The paper notes that current models can synthesize plausible futures that violate physical laws.Source: §6.2 · p. 21
  • Future embodied VLA research should target lightweight architectures, efficient tokenization, and uncertainty quantification to address latency, interpretability, and trustworthy deployment.The paper highlights the millisecond-level reaction requirements of autonomous systems.Source: §6.2 · p. 21
  • Future 4D semantic-geometric unification should add dense semantic and instance attributes over time to continuous Gaussian primitives, enabling interaction with specific objects in dynamic scenes.The paper identifies spatiotemporal consistency in Gaussian attributes as a pivotal pre-training task.Source: §6.2 · p. 21
  • Future long-tail safety systems should add logical deduction and counterfactual causal reasoning to pattern-recognition paradigms, allowing them to override reactive policies in novel hazards.The paper identifies Chain-of-Thought distillation from LLMs as a possible route to System 2 capabilities.Source: §6.2 · p. 21

7 Conclusion

The paper concludes that multi-modal pre-training progresses from modality-specific methods toward unified foundation models, with camera–LiDAR integration producing semantically rich and geometrically precise representations. It identifies generative, embodied, and reasoning capabilities as future directions for robust, scalable deployment.

7 Conclusion

  • Camera–LiDAR integration creates representations that are both semantically rich and geometrically precise.The analysis characterizes unified foundation models as the cornerstone of Spatial Intelligence.Source: §7 · p. 21
  • Pre-trained foundation models are described as essential for open-world generalization and mitigating scarce annotated 3D data.Source: §7 · p. 21
  • Generative World Models acting as neural simulators and Vision-Language-Action frameworks unifying perception with decision-making are proposed as future directions.The paper frames these as ways to bridge the semantic-geometric gap and connect perception with decisions.Source: §7 · pp. 21–22
  • Explicit reasoning is identified as pivotal for handling long-tail unpredictability in real-world environments.Source: §7 · p. 21
  • Further generative and embodied pre-training is presented as instrumental to robust, scalable autonomous systems for safe real-world deployment.Source: §7 · p. 22

Related papers

Loading 2512.24385v2…