DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset

• 6G systems increasingly combine communication, multimodal sensing, and positioning, creating research opportunities for machine learning.Source: §I · p. 1
• Synthetic datasets such as DeepMIMO and ViWi support initial algorithm development but may not reveal performance in real-world deployments.Source: §I · p. 1
• More than 1 million real-world data points provide synchronized communication and multimodal sensing measurements across 40+ diverse scenarios.Source: §I · p. 1
• The article describes the dataset structure, testbeds, collection methodology, scenarios, and applications to facilitate adoption and reproducibility.Source: §I · p. 1

• Effective learning for communication and sensing requires co-existing measurements synchronized closely enough to reflect the same system state.The example requirement is that wireless communication and sensing measurements be collected within the same coherence time.Source: §II · p. 1
• The dataset should combine modalities such as visual data, LiDAR, GPS positions, and weather measurements with communication data.Source: §II · p. 1
• Real-world measurements are needed because practical imperfections are difficult to model and capture in synthetic datasets.Source: §II · p. 1
• A large-scale dataset with sufficient measurement variance supports solutions that are scalable and robust to distribution shifts.Source: §II · p. 2
• DeepSense 6G addresses these objectives with real-world, co-existing multimodal communication and sensing data covering varied use cases.Source: §II · p. 2

• Each scenario is a standalone dataset containing multimodal sensing and communication data from one typically long collection session.Source: §III · p. 2
• A unified format across scenarios facilitates combining data from multiple sites and times to build larger, more diverse datasets.Source: §III · p. 2
• The scenario-based structure allows the dataset to grow by adding scenarios for additional use cases.Source: §III · p. 2
• The unified structure simplifies dataset access, definition, and reproducibility.Source: §III · p. 2
• Data from sensors and units is collected and synchronized to the same sampling interval within each scenario.Source: §III · p. 2

• Each modular testbed contains units equipped with wireless and environmental sensors plus processing hardware for configuration and storage.Sensors may include wireless transceivers, RGB cameras, 2D/3D LiDAR, radar, and GPS RTK kits.Source: §IV · p. 2
• Testbeds are selected and configured for targeted applications, enabling coverage of multiple deployment scenarios.Source: §IV · p. 2
• Fig. 2 organizes scenario generation into real-world data collection, data processing, and data filtering.Source: Figure 2 · p. 3
• Testbed 5 targets vehicle-to-infrastructure applications with a stationary basestation unit and a vehicle unit.Source: §IV · p. 3
• Testbed 5 includes 60GHz mmWave reception, RGB imaging, 3D LiDAR, FMCW radar, and RTK GPS sensing.Source: §IV · p. 3
• At each time instant, the vehicle-side data includes GPS position, RGB imagery, radar I/Q samples, LiDAR point clouds, and 64-element beam-training power vectors.Source: §IV · p. 3

• Each DeepSense 6G scenario consists of multi-modal sensing and communication data collected during one long data-collection session.Source: §V · p. 3
• Generating a new scenario comprises two main phases: data collection and data processing.Source: §V · p. 3
• The framework includes multiple steps for generating the final development dataset.Source: §V · p. 3

• Data collection comprises two phases: off-field planning and on-field collection.Planning defines objectives, required units and sensors, and the collection time and location before deployment.Source: A. Data Collection · p. 3
• Sensor field-of-view alignment calibrates mmWave, camera, LiDAR, and radar sensors before collection.In testbed 5, the RGB camera has a 110° field of view, whereas the phased array has 90°.Source: A. Data Collection · p. 3
• Cross-modality capture-rate alignment is necessary because sensing elements operate at different rates, including 10 Hz, 30 Hz, and 20 sweeps per second.Source: A. Data Collection · p. 3
• The dataset includes 40+ deployment scenarios spanning vehicle-to-infrastructure, vehicle-to-vehicle, drone, RIS, pedestrian, and indoor use cases.Source: Figure 3 · p. 4
• Real-time monitoring uses manual checks and data plotting to assess collected-data quality and correctness.Source: A. Data Collection · p. 4

• Raw data undergoes post-processing that includes data synchronization and filtering to generate the final development dataset.Source: B. Data Processing · p. 4
• UTC timestamps align modalities with different capture rates, using recorded time as the synchronization pivot.Source: B. Data Processing · p. 4
• Scenario-dependent filtering removes samples outside the desired field of view, such as mobile users outside a basestation’s view in V2I data.Source: B. Data Processing · p. 4
• In-house verification tools visualize synchronized and filtered modalities together to identify synchronization issues, missing data, and corrupted data.Source: B. Data Processing · p. 4

Scenario diversity

• DeepSense 6G contains 40+ realistic, continuously growing scenarios collected as long data-collection sessions across multiple locations.Source: §VI · p. 4

Multiple sensors

• Data fusion combines modalities such as RGB cameras, 2D/3D LiDAR, radar, and GPS RTK to address individual sensor limitations.Source: §VI · pp. 4–5

Multiple locations

• Scenarios were collected across two countries and more than 15 indoor and outdoor locations.Source: §VI · p. 5

Environmental and user diversity

• Lighting, weather, object classes, instances, and speeds introduce variance relevant to sensing-aided beam prediction, blockage prediction, and handoff.Source: §VI · pp. 5–6

Example application

• Figure 4 compares beam-index prediction trained and tested within scenario 5 against training on scenario 1 and testing on scenario 5.The cross-scenario case evaluates whether the solution predicts optimal beam indices without scenario 5 training data.Source: Figure 4 · p. 6

Deployment scenarios

• The scenarios cover vehicle-to-infrastructure, vehicle-to-vehicle, drone, RIS, indoor, and pedestrian deployments.Source: §VI · p. 6

Enabled applications

• The dataset supports applications including beam and channel prediction, blockage prediction and hand-off, ISAC waveform design, radar/LiDAR object detection and classification, and mmWave positioning.Source: §VII · p. 6

Task-specific datasets

• Task-specific development datasets accelerate benchmarking, reproducibility, and comparisons for key machine learning applications.Source: §VII · p. 6
• The unified data structure allows task-specific datasets to combine data from multiple scenarios.Source: §VII · p. 6

Generalizability evaluation

• Scenarios 1 and 5 provide wireless, visual, and position data collected across locations, times, and lighting conditions for evaluating generalizability.Source: §VII · p. 6
• More than 70% top-3 accuracy was achieved when training on scenario 1 and evaluating on unseen scenario 5 data.Source: §VII · p. 7
• Incorrect beam predictions were often close to the optimal beams and could still provide good receive power.Source: §VII · p. 7

• DeepSense 6G comprises synchronized multi-modal sensing and communication data from more than 40 deployment scenarios.Source: §VIII · p. 7
• The article describes the dataset’s scalable structure, modular testbed definition, available scenarios, and machine learning applications.Source: §VIII · p. 7
• The project aims to support reproduction and benchmarking of datasets and research results in communication-and-sensing applications.Source: §VIII · p. 7