Source-linked AI summary
DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset
Ahmed Alkhateeb, Gouranga Charan, Tawfik Osman, Andrew Hredzak, João Morais, Umut Demirhan, Nikhil Srinivas
TL;DR
Machine-learning research for communication, sensing, and positioning needs large-scale real-world data because synthetic datasets may not capture deployment imperfections. DeepSense 6G provides synchronized multimodal measurements in a scalable scenario-based framework, documenting its testbeds and processing to support reproducible research across diverse applications. The dataset includes 40+ real-world scenarios and has been used to evaluate beam prediction, achieving more than 70% top-3 accuracy when transferring from scenario 1 to scenario 5 without scenario-5 training data.
Problem
Synthetic datasets support initial development but may not provide realistic evidence for machine-learning performance in real-world deployments.
Method
The article constructs and documents a scalable dataset of synchronized real-world multimodal sensing and communication data organized into unified standalone scenarios and modular testbeds.
Results
The dataset contains more than 40 real-world deployment scenarios, and cross-scenario beam prediction achieved more than 70% top-3 accuracy without training data from scenario 5.
Takeaways & Limitations
DeepSense 6G supports deep-learning research and reproducible benchmarking across applications involving communication, sensing, and positioning.
Abstract
from arXiv · showhide
This article presents the DeepSense 6G dataset, which is a large-scale dataset based on real-world measurements of co-existing multi-modal sensing and communication data. The DeepSense 6G dataset is built to advance deep learning research in a wide range of applications in the intersection of multi-modal sensing, communication, and positioning. This article provides a detailed overview of the DeepSense dataset structure, adopted testbeds, data collection and processing methodology, deployment scenarios, and example applications, with the objective of facilitating the adoption and reproducibility of multi-modal sensing and communication datasets.
I. INTRODUCTION
DeepSense 6G addresses the need for realistic, large-scale data for learning at the intersection of communication, sensing, and positioning. It provides more than 1 million real-world, synchronized multimodal measurements across 40+ deployment scenarios and documents the dataset for adoption and reproducibility.
- 6G systems increasingly combine communication, multimodal sensing, and positioning, creating research opportunities for machine learning.
- Synthetic datasets such as DeepMIMO and ViWi support initial algorithm development but may not reveal performance in real-world deployments.
- More than 1 million real-world data points provide synchronized communication and multimodal sensing measurements across 40+ diverse scenarios.
- The article describes the dataset structure, testbeds, collection methodology, scenarios, and applications to facilitate adoption and reproducibility.
II. WHY DEEPSENSE 6G?
DeepSense 6G is motivated by the need for synchronized, multimodal, real-world, large-scale data that can support learning across changing wireless scenarios. Its scalable dataset design targets diverse use cases and applications spanning communications, sensing, and positioning.
- Effective learning for communication and sensing requires co-existing measurements synchronized closely enough to reflect the same system state.The example requirement is that wireless communication and sensing measurements be collected within the same coherence time.
- The dataset should combine modalities such as visual data, LiDAR, GPS positions, and weather measurements with communication data.
- Real-world measurements are needed because practical imperfections are difficult to model and capture in synthetic datasets.
- A large-scale dataset with sufficient measurement variance supports solutions that are scalable and robust to distribution shifts.
- DeepSense 6G addresses these objectives with real-world, co-existing multimodal communication and sensing data covering varied use cases.
III. DEEPSENSE DATASET STRUCTURE
DeepSense 6G organizes data as standalone scenarios collected in long sessions and represented through a unified format. This structure supports dataset growth, cross-scenario combination, access, and reproducibility.
- Each scenario is a standalone dataset containing multimodal sensing and communication data from one typically long collection session.
- A unified format across scenarios facilitates combining data from multiple sites and times to build larger, more diverse datasets.
- The scenario-based structure allows the dataset to grow by adding scenarios for additional use cases.
- The unified structure simplifies dataset access, definition, and reproducibility.
- Data from sensors and units is collected and synchronized to the same sampling interval within each scenario.
IV. DEEPSENSE TESTBEDS
DeepSense 6G uses modular testbeds composed of self-sustained sensor-equipped units to support diverse scenarios and applications. The collection workflow proceeds from real-world acquisition through processing and filtering into development datasets.
- Each modular testbed contains units equipped with wireless and environmental sensors plus processing hardware for configuration and storage.Sensors may include wireless transceivers, RGB cameras, 2D/3D LiDAR, radar, and GPS RTK kits.
- Testbeds are selected and configured for targeted applications, enabling coverage of multiple deployment scenarios.
- Fig. 2 organizes scenario generation into real-world data collection, data processing, and data filtering.
- Testbed 5 targets vehicle-to-infrastructure applications with a stationary basestation unit and a vehicle unit.
- Testbed 5 includes 60GHz mmWave reception, RGB imaging, 3D LiDAR, FMCW radar, and RTK GPS sensing.
- At each time instant, the vehicle-side data includes GPS position, RGB imagery, radar I/Q samples, LiDAR point clouds, and 64-element beam-training power vectors.
V. DATA COLLECTION AND PROCESSING FRAMEWORK
The DeepSense 6G dataset is organized into scenarios, each built from a long data-collection session and processed through data collection and data processing phases.
- Each DeepSense 6G scenario consists of multi-modal sensing and communication data collected during one long data-collection session.
- Generating a new scenario comprises two main phases: data collection and data processing.
- The framework includes multiple steps for generating the final development dataset.
A. Data Collection
Data collection combines off-field planning with on-field testbed deployment, sensor alignment, and real-time quality monitoring across diverse deployment scenarios.
- Data collection comprises two phases: off-field planning and on-field collection.Planning defines objectives, required units and sensors, and the collection time and location before deployment.
- Sensor field-of-view alignment calibrates mmWave, camera, LiDAR, and radar sensors before collection.In testbed 5, the RGB camera has a 110° field of view, whereas the phased array has 90°.
- Cross-modality capture-rate alignment is necessary because sensing elements operate at different rates, including 10 Hz, 30 Hz, and 20 sweeps per second.
- The dataset includes 40+ deployment scenarios spanning vehicle-to-infrastructure, vehicle-to-vehicle, drone, RIS, pedestrian, and indoor use cases.
- Real-time monitoring uses manual checks and data plotting to assess collected-data quality and correctness.
B. Data Processing
Data processing transforms raw measurements into a final development dataset through synchronization, scenario-dependent filtering, and verification of multimodal data quality.
- Raw data undergoes post-processing that includes data synchronization and filtering to generate the final development dataset.
- UTC timestamps align modalities with different capture rates, using recorded time as the synchronization pivot.
- Scenario-dependent filtering removes samples outside the desired field of view, such as mobile users outside a basestation’s view in V2I data.
- In-house verification tools visualize synchronized and filtered modalities together to identify synchronization issues, missing data, and corrupted data.
VI. DEEPSENSE 6G SCENARIOS
DeepSense 6G provides diverse real-world multimodal scenarios across deployments, sensing modalities, locations, times, weather conditions, and user equipment, supporting varied wireless applications.
- Scenario diversity: DeepSense 6G contains 40+ realistic, continuously growing scenarios collected as long data-collection sessions across multiple locations.
- Multiple sensors: Data fusion combines modalities such as RGB cameras, 2D/3D LiDAR, radar, and GPS RTK to address individual sensor limitations.
- Multiple locations: Scenarios were collected across two countries and more than 15 indoor and outdoor locations.
- Environmental and user diversity: Lighting, weather, object classes, instances, and speeds introduce variance relevant to sensing-aided beam prediction, blockage prediction, and handoff.
- Example application: Figure 4 compares beam-index prediction trained and tested within scenario 5 against training on scenario 1 and testing on scenario 5.The cross-scenario case evaluates whether the solution predicts optimal beam indices without scenario 5 training data.
- Deployment scenarios: The scenarios cover vehicle-to-infrastructure, vehicle-to-vehicle, drone, RIS, indoor, and pedestrian deployments.
VII. ENABLED APPLICATIONS AND ML TASKS
The dataset supports diverse communication, sensing, and positioning applications, including cross-scenario machine learning evaluation. Its unified structure enables task-specific datasets and studies of generalizability, transfer learning, robustness, scalability, and distribution shift.
- Enabled applications: The dataset supports applications including beam and channel prediction, blockage prediction and hand-off, ISAC waveform design, radar/LiDAR object detection and classification, and mmWave positioning.
- Task-specific datasets: Task-specific development datasets accelerate benchmarking, reproducibility, and comparisons for key machine learning applications.
- Task-specific datasets: The unified data structure allows task-specific datasets to combine data from multiple scenarios.
- Generalizability evaluation: Scenarios 1 and 5 provide wireless, visual, and position data collected across locations, times, and lighting conditions for evaluating generalizability.
- Generalizability evaluation: More than 70% top-3 accuracy was achieved when training on scenario 1 and evaluating on unseen scenario 5 data.
- Generalizability evaluation: Incorrect beam predictions were often close to the optimal beams and could still provide good receive power.
VIII. CONCLUSION
DeepSense 6G is a large-scale real-world dataset of synchronized multi-modal sensing and communication data spanning more than 40 deployment scenarios. The article describes its structure, testbeds, scenarios, and applications to support reproducible benchmarking and research.
- DeepSense 6G comprises synchronized multi-modal sensing and communication data from more than 40 deployment scenarios.
- The article describes the dataset’s scalable structure, modular testbed definition, available scenarios, and machine learning applications.
- The project aims to support reproduction and benchmarking of datasets and research results in communication-and-sensing applications.