Source-linked AI summary
uScenes: A Multimodal RGB and 3D Sonar Dataset for Underwater Robot Perception
Trung Tien Dong, Zhenqi Wu, Aditya Penumarti, Zi-Hao Zhang, Micaiah Bartlett, Jane Shin, Xiaomin Lin
TL;DR
Underwater perception needs sensing that remains useful when optical imagery degrades, while conventional forward-looking sonar leaves elevation unresolved. uScenes introduces a multimodal field dataset pairing synchronized RGB images with 3D multibeam sonar point clouds. It contains 110 scenes and 95,834 synchronized observations representing 277.6 minutes of recordings, providing a foundation for underwater sensor fusion and 3D scene understanding.
Problem
Optical cameras degrade under poor visibility, while forward-looking sonar measures range and bearing without resolving elevation, complicating 3D localization and scene understanding.
Method
The paper constructs uScenes, a multimodal underwater dataset of synchronized RGB images and metric 3D multibeam sonar point clouds with associated metadata and coordinate conventions.
Results
uScenes contains 110 scenes, 95,834 synchronized observations, and 277.6 minutes of underwater recordings.
Takeaways & Limitations
By pairing optical appearance with directly measured acoustic geometry, uScenes provides a foundation for underwater sensor fusion, cross-modal representation learning, and 3D scene understanding.
Takeaways & Limitations
The dataset was collected with one robotic platform at one freshwater site, so it does not establish generalization across sensors, platforms, locations, or saltwater environments.
Abstract
from arXiv · showhide
Robust perception is essential for the deployment of autonomous underwater robots. However, optical cameras become unreliable under poor illumination and backscatter. Forward looking (2D) acoustic sensors remain effective under these conditions, but they measure range and bearing while leaving elevation unresolved, creating an ambiguity that prevents individual sonar returns from being localized in three dimensional (3D) space. This complicates the sensor use for 3D scene understanding and precise object detection. We introduce \textbf{uScenes}, a multimodal underwater dataset containing synchronized 3D multibeam sonar point clouds and RGB imagery. The dataset contains 110 scenes and 95,834 synchronized observation, representing 277.6 minutes of data collected across multiple field sessions. uScenes establishes a foundation for underwater sensor fusion, cross modal representation learning and 3D scene understanding. Code and datasets are given at https://github.com/era-research-lab/uScenes.
I. INTRODUCTION
Underwater robots need perception that remains useful when optical imagery degrades, but conventional forward-looking sonar leaves elevation unresolved. uScenes addresses this gap with synchronized RGB imagery and metric 3D multibeam sonar observations collected in field environments.
- Optical cameras can degrade under poor illumination, suspended particles, changing water conditions, and backscatter, despite providing rich appearance and semantic information.The degradation is associated with light attenuation, color distortion, and backscatter.
- Forward-looking sonar measures range and bearing but leaves elevation unresolved, so returns may correspond to multiple 3D positions.This ambiguity complicates 3D scene understanding and precise spatial localization of detected objects.
- 3D multibeam sonar measures horizontal and vertical angular dimensions, allowing valid range observations to map directly to 3D Cartesian points.Compact multibeam systems can produce volumetric observations in real time from a moving platform.
- Existing datasets pair optical data with 2D sonar imagery, provide sonar-derived 3D point clouds without synchronized RGB, or obtain 3D geometry from laser scanners rather than sonar.These distinctions motivate a dataset combining synchronized RGB imagery with directly measured 3D sonar geometry.
- uScenes contains 110 scenes, 95,834 synchronized RGB and 3D multibeam sonar observations, and 277.6 minutes of recordings from a mobile robotic platform.The dataset also includes metric sonar points, normalized signal strength, timestamps, metadata, and coordinate conventions.
- The dataset construction covers the sensing platform, temporal association, point-cloud generation, data organization, and optic-acoustic calibration procedures.
A. Underwater Perception Modalities
Underwater perception combines optical and acoustic sensing because the modalities provide complementary information under changing visibility. Existing optical and acoustic datasets support important tasks, but many acoustic observations remain two dimensional or lack simultaneous optical measurements.
- Optical cameras provide color, texture, and appearance, while acoustic sensors remain usable when attenuation, scattering, turbidity, or insufficient illumination degrade images.
- Acoustic images contain speckle, weak texture, and sensor-dependent distortions that complicate conventional visual feature detection.
- Most imaging sonars encode range and bearing without directly resolving elevation, leaving individual returns ambiguous in 3D space.Prior work addressed this through motion estimation, acoustic structure from motion, and reconstruction from multiple sonar views.
- Optical datasets provide substantial appearance diversity and annotations but do not include acoustic measurements available when optical visibility deteriorates.USIS16K contains 16,151 images spanning 158 object categories.
- Acoustic datasets support classification, detection, segmentation, and feature learning, but their observations may remain two dimensional and unpaired with simultaneous optical measurements.UATD contains more than 9,000 multibeam forward-looking sonar images, while marine-debris datasets span tank, turntable, and flooded-quarry settings.
C. Multimodal Optic Acoustic Datasets
Prior work studied optic-acoustic geometry, synchronized multimodal recordings, simulation, and sonar point clouds, but uScenes combines these directions in a mobile field dataset with metric 3D acoustic returns.
- Prior optic-acoustic studies developed epipolar geometry, joint calibration, 3D reconstruction, and refraction-aware models for corresponding optical and sonar measurements.
- RGBS50 and R-S9 provide synchronized optical-acoustic observations, while HODOR offers long-duration stereo-camera and sonar video from a stationary fish observatory.These datasets target tracking, fusion classification, fish monitoring, biomass estimation, or behavioral analysis.
- Simulation platforms provide repeatable and configurable multimodal experiments, while MAOUD combines RGB, acoustic imagery and video, and laser point clouds in a controlled environment.MAOUD’s geometric information comes from laser point clouds rather than acoustic point clouds.
- BrSPCD and SUOP demonstrate the value of underwater sonar point clouds for segmentation, object recognition, and mapping, while prior BlueROV2 work showed scan registration and calibration.
- uScenes provides a scene-organized collection of synchronized RGB images and compact 3D sonar point clouds acquired from a mobile underwater robot.Its acoustic modality supplies metric 3D returns for every synchronized observation.
- Pairing optical appearance with acoustic geometry supports research connecting visual information and acoustic geometry across dynamic field recordings.
III. DATASET
The uScenes platform rigidly mounts an RGB camera and 3D multibeam sonar on a BlueROV2 for synchronized underwater data collection. The resulting observations combine RGB imagery with registered sonar returns across varied field environments.
- The platform uses a BlueROV2 equipped with a DWE underwater RGB camera and Water Linked Sonar 3D-15 in a rigid, forward-facing configuration.The sonar is near the vehicle center and the laterally positioned camera has an overlapping forward field of view.
- The camera records 1280 × 720 images at approximately 30 Hz, while the sonar produces 256 × 64 range observations at approximately 6 Hz.
- Each valid sonar measurement provides range and registered signal strength, from which a 3D point cloud is constructed.A sonar frame contains between 1 and 12,627 returns, with an average of 6,805 returns.
- Data were collected at Blue Grotto in Williston, Florida, during seven field sessions across natural and human-influenced underwater environments.Observed content includes divers, fish, underwater robots, and submerged structures.
B. Temporal Association
uScenes pairs each sonar observation with the nearest camera image by timestamp, subject to an 83 ms acceptance threshold. The selected pair is intended to align the image with the midpoint of the sonar scan, while within-scan motion remains possible.
- Each sonar observation is paired with the camera image having the nearest timestamp.
- 83 ms is the maximum allowed timestamp difference for accepting a synchronized observation pair.This threshold is approximately half of one sonar acquisition interval.
- 13.2 ms mean and 11.2 ms median absolute timestamp differences characterize the accepted observation pairs.95 percent of pairs differ by no more than 32.8 ms.
- The sonar timestamp represents the midpoint of its sequential scan, so pairing uses the camera image nearest that scan-center time.Motion occurring within a sonar scan remains present.
D. Dataset Organization
Each uScenes scene stores paired camera images, sonar point clouds, and timestamp metadata, with manifests documenting pair-level timing and return counts. Scene-level descriptions provide coarse recording categories rather than frame-level object annotations.
- Each scene contains paired JPEG camera images, binary sonar point clouds, and timestamp metadata.Each sonar row stores [x,y,z,s], where s is normalized acoustic signal strength.
- The JSON Lines manifest records camera and sonar timestamps, their difference, frame identifiers, and sonar-return counts for every pair.
- A global scene index records session information, duration, and coarse scene-level task descriptions.These descriptions identify primary recording content but are not frame-level detection annotations.
- 54 of 110 scenes contain human or structure recordings, while 46 contain no designated object of interest.These categories are intended for selecting recordings rather than serving as object annotations.
E. Optic Acoustic Calibration Methodology
Optic-acoustic calibration estimates a shared rigid transformation between sonar and camera frames from corresponding 3D sonar points and 2D image locations. The methodology calibrates camera intrinsics underwater and uses multiple target configurations before robust nonlinear refinement.
- Calibration establishes the spatial relationship between camera and sonar using selected 3D sonar points and corresponding 2D image locations.
- Camera intrinsics are estimated underwater from multiple observations of an 8 × 8 white pin circle grid with radial and tangential distortion.The underwater procedure accounts for imaging effects from the camera housing and water interface.
- Three target configurations provide identifiable sonar 3D points and corresponding camera 2D locations: dimension-constrained concrete blocks, round-headed pin boards, and flat-headed screw boards.
- Flat screw heads produce more isolated acoustic responses than round-headed pins, making individual reflectors easier to identify in the 3D point cloud.Two screw boards are positioned at different depths, with board identity and grid position preserving correspondence labels.
- The rigid sonar-to-camera transformation is estimated by minimizing reprojection error, initialized with EPnP and nonlinearly refined over all selected correspondences.Because the sensors are rigidly mounted, all target configurations and observations share one rotation and translation.
A. Research Opportunities
uScenes pairs synchronized RGB appearance with 3D sonar geometry to support underwater multimodal perception research. Its temporal pairing supports cross-modal learning, while spatial registration remains a prerequisite for point-to-image tasks.
- Synchronized RGB images and sonar point clouds provide complementary appearance and three-dimensional structure for underwater perception research.The combination supports sensor fusion, cross-modal representation learning, and multimodal 3D scene understanding.
- Temporal synchronization enables models to learn relationships between modalities without requiring direct point-to-pixel correspondence.
- Precise spatial association requires an accurate transformation between sonar and camera coordinate frames.The calibration procedure considers targets, correspondence construction, and optimization for the sensing platform.
- Temporally synchronized observations can support temporal correspondence, scene-level association, and independent modality processing even without spatial registration.
- Projecting individual sonar points into camera images requires establishing and validating an appropriate optic-acoustic calibration.The included calibration sequences provide material for developing and comparing such methods.
C. Limitations
uScenes provides a substantial synchronized RGB and 3D sonar collection, but its current scope and annotations constrain the benchmarks and generalization it can establish. Calibration, annotation expansion, and broader data collection remain future needs.
- The dataset uses one robotic platform at a single freshwater site, so it does not establish generalization across sensors, platforms, locations, or saltwater environments.
- Dense object annotations are absent, and scene descriptions are not frame-level ground truth for supervised detection or segmentation benchmarks.Additional 2D and 3D annotations are required to establish those benchmarks.
- The released acoustic signal strength is relative rather than calibrated reflectivity and should not be directly compared across sonar frames.It is compensated for range and normalized independently within each sonar frame.
- 95,834 paired observations across 110 scenes represent 277.6 minutes of recordings from seven field sessions.The paper also describes the sensing platform, temporal association, sonar point-cloud construction, organization, and calibration methods.
- Future work will establish reliable optic-acoustic calibration, expand annotations, and collect data across additional platforms and underwater environments.