PaperlayerView exact revision on arXiv

Source-linked AI summary

RSFusionDet: Underwater RGB-Sonar Multimodal Object Detection

Zhuoyan Liu, Yihan Wang, Bo Wang, Bing Wang, Ye Li

arXiv:2608.25367v1cs.CV

TL;DR

Underwater RGB and sonar sensing have complementary strengths, but their different imaging characteristics make multimodal object detection and correspondence challenging. The paper introduces the RSFusion benchmark and RSFusionDet, combining cross-attention fusion with an object-matching head and loss. On RSFusion, the method achieves 76.4/48.6 AP (RGB/Sonar) and 83.4 F1-Scorematch for object matching, while reported conclusions identify remaining opportunities for further research.

  • Problem

    RGB and sonar complement each other underwater, but RGB-Sonar detection must address spatially misaligned fusion, heterogeneous-space prediction, and cross-modal object association.

  • Method

    The paper creates the 7073-pair RSFusion benchmark and proposes RSFusionDet with CAFusion for feature fusion and OMHead with OMLoss for cross-modal object matching.

  • Results

    76.4/48.6 AP (RGB/Sonar) and 83.4 F1-Scorematch are achieved on RSFusion for object detection and object matching.

  • Takeaways & Limitations

    The benchmark and model support RGB-Sonar multimodal object detection with separate RGB and sonar detection results linked by object matching.

  • Takeaways & Limitations

    The authors report remaining RGB-Sonar object-mismatch errors and state that further research is needed beyond the RSFusion benchmark.

Abstract

from arXiv · show

Underwater unimodal object detection faces many challenges in sensor imaging, such as optical images limited by underwater noise and visible distance, and sonar images limited by less object structural information. While, optical images have rich object structural information, and sonar images are less affected by underwater noise and have a longer visible distance. Optical (RGB modality) and sonar (Sonar modality) images have complementary information underwater. In this paper, we create an RGB-Sonar multimodal object detection dataset, \textbf{R}GB-\textbf{S}onar \textbf{Fusion} (RSFusion) and propose evaluation metrics for the benchmark. And we propose the \textbf{R}GB-\textbf{S}onar \textbf{Fusion} \textbf{Det}ector (RSFusionDet) with a new RGB-Sonar multimodal object detection result expression for RGB-Sonar multimodal object detection. We analyze the features of RGB and Sonar modal information, and design a Cross-Attention Fusion (CAFusion) module to fuse RGB-Sonar spatial misalignment features and Object Matching Head (OMHead) with Loss (OMLoss) to match identical objects in RGB-Sonar modalities. Our RSFusionDet achieves 76.4/48.6 AP (RGB/Sonar) for object detection and 83.4 \(\text{F1-Score}_{match}\) for object matching, on RSFusion, which outperforms other object detection models. Compared with the DINO baseline, our method improves by 0.7/1.4 AP (RGB/Sonar) while simultaneously providing reliable cross-modal object matching. The code and datasets are publicly available at https://github.com/LEFTeyex/RSFusionDet.

1. Introduction

The paper frames underwater RGB-Sonar detection as a multimodal problem because RGB and sonar provide complementary information but suffer different sensing limitations. It introduces the RSFusion benchmark and RSFusionDet, which separately fuse and detect in both feature spaces before matching corresponding objects.

  • Motivation: RGB images provide rich color, texture, shape, and edge information but are limited underwater by light attenuation, color cast, blur, and short visible distance.Optical visibility is typically 10–30 meters and may fall to 5–10 meters or less in deep or turbid water.
  • Motivation: Sonar is less affected by underwater visibility conditions but provides poorer structural information, making object-category recognition difficult.The paper motivates combining sonar's sensing range with RGB's richer object features.
  • Research challenges: RGB-Sonar detection introduces spatially misaligned feature fusion, independent predictions in heterogeneous spaces, and cross-modal association of multiple detected objects.RGB and sonar depict different spatial planes, such as front-view RGB and top-view sonar imagery.
  • Contributions: RSFusion provides a benchmark with 7073 RGB-Sonar image pairs and evaluation metrics, while RSFusionDet uses a dual-branch architecture for multimodal detection.The model introduces CAFusion for feature fusion and OMHead with OMLoss for matching identical objects across modalities.
  • Results: 76.4/48.6 AP (RGB/Sonar) and 83.4 F1-Scorematch are achieved for detection and object matching on RSFusion.The reported results are supported by extensive ablation studies and visualization results.

2. Related Work

Related work spans unimodal one-stage and two-stage detectors, newer NMS-free detectors, and multimodal fusion methods across several sensor combinations. Existing fusion approaches often exploit aligned modalities or transform features into a shared representation.

  • Unimodal object detection: Two-stage detectors use region proposals followed by refinement, whereas one-stage detectors directly output object predictions.The cited examples include R-CNN and SPP-Net for two-stage detection, and YOLO, SSD, RetinaNet, FCOS, ATSS, and TOOD for one-stage detection.
  • Unimodal object detection: NMS-free detectors such as DINO remove nonmaximum suppression from conventional detection pipelines and can also be applied to underwater unimodal detection.
  • Multimodal object detection: Multimodal object detection research includes RGB-Thermal, RGB-Event, RGB-LiDAR, and RGB-Radar fusion methods.The cited approaches use attention, cross-attention, depth guidance, or BEV representations for multimodal feature fusion.

3. RSFusionDet

RSFusionDet uses cross-attention to align and fuse RGB-Sonar features separately in RGB and Sonar spaces, then performs modality-specific detection and cross-modal object matching. Its architecture combines CAFusion for fusion with OMHead and OMLoss for correspondence learning.

  • Overall architecture: RSFusionDet extracts RGB and Sonar features with separate identical backbones and processes fused features through individual detection pipelines.The pipelines use DINO encoders, decoders, and training methods to produce RGB and Sonar detection results.
  • Cross-Attention Fusion: CAFusion uses multi-scale deformable cross-attention to align Sonar features with RGB features and RGB features with Sonar features.The module performs alignment and fusion independently in the two modality feature spaces.
  • Cross-Attention Fusion: CAFusion concatenates aligned cross-modal features with original modality features, then applies a linear layer, Layer Normalization, and ReLU activation.In RGB space, aligned Sonar features are concatenated with RGB features; the Sonar-space operation is identical.
  • Cross-Attention Fusion: Cross-deformable attention supports global multimodal alignment and cross-scale fusion while reducing computational complexity.The paper characterizes CAFusion as having a simple structure and low computational complexity.
  • Object Matching Head and Loss: OMHead computes cosine similarities between RGB and Sonar decoder matching features to establish identical-object correspondences.It also incorporates object bounding-box prior information, while OMLoss uses an IoU weight formed from RGB and Sonar prediction-to-ground-truth IoUs.
  • Object Matching Head and Loss: OMHead filters matching features to reduce computational complexity during training and inference before producing one-to-one object matches.Training uses Hungarian-loss matches for filtering, while inference retains detections above a 0.3 score threshold; matching uses a 0.8 cosine-similarity threshold.

4. Datasets and Evaluation Metrics

RSFusion is an underwater RGB-Sonar multimodal object detection dataset with paired data across object categories, lighting conditions, and sonar ranges. The paper defines separate detection and cross-modal object-matching metrics for evaluating these capabilities.

  • Dataset: RSFusion contains seven object classes and paired RGB-Sonar images captured simultaneously with a camera and a 1.2 MHz 2D sonar.The classes are mannequin, UUV, reflector, metal polyhedron, float, iron ball, and frustum.
  • Dataset: The dataset includes paired images from light and dark scenes and maximum sonar ranges of 5, 10, 15, and 20 meters.Maximum sonar range denotes the maximum range at which the sonar sensor scans and images.
  • Evaluation Metrics: Evaluation separates RGB and Sonar object detection metrics from RGB-Sonar object-matching metrics.Detection uses AP for each modality, while matching uses Pmatch, Rmatch, and F1-Scorematch.
  • Evaluation Metrics: Detection evaluation follows COCO AP reporting and additionally measures light/dark scenes and Sonar performance at different maximum ranges.Reported scene metrics include APlight and APdark; range metrics include APsr5 and APsr10 among the defined settings.
  • Evaluation Metrics: Object matching assigns predicted pairs using maximum IoU against ground truth, removes pairs with IoU below zero, and counts correct matches as N.Pmatch = N/Npred and Rmatch = N/Ngt, with F1-Scorematch derived from precision and recall.

5.1. Implementation Details

RSFusionDet is trained as a paired dual-stream RGB-Sonar system, while comparisons use a separate detection framework to maintain consistency with unimodal baselines. Experiments use standardized training settings and compare models with and without the proposed fusion and matching components.

  • Framework and Comparisons: RSFusionDet uses paired RGB and Sonar inputs throughout training and testing, with separate modality-specific detector pipelines for fair unimodal comparisons.The separate framework contains two independent unimodal object detectors for RGB and Sonar.
  • Dataset Comparison: RSFusion is more diverse than RGBS50 because it annotates all visible objects, category labels, modality-specific boxes, and cross-modal correspondence labels.RGBS50 is described as a single-object tracking dataset with substantially less object diversity.
  • Framework and Comparisons: The comparison includes state-of-the-art unimodal models without RGB-Sonar feature fusion and object matching.Table 1 reports RGB/Sonar detection AP, while other models lack object-matching metrics.
  • Framework and Comparisons: Table 2 evaluates adding CAFusion and OMHead to other models, with RSFusionDet defined as DINO+CA+OM.CA denotes CAFusion and OM denotes OMHead.

5.2. Comparisons with State-of-the-arts

Across RSFusion comparisons, RSFusionDet improves RGB and Sonar detection while enabling cross-modal object matching, with gains observed across object classes and sonar-range settings. Visualizations support its fusion behavior, but small occluded objects and mismatched cross-modal pairs remain difficult.

  • Comparisons with Other Models: 76.4/48.6 AP (RGB/Sonar AP) is achieved by RSFusionDet, improving 0.7/1.4 AP over the best DINO baseline.The method outperforms the compared models and adds only 3 M parameters and 16 GFLOPs relative to DINO.
  • Comparisons with Other Models: RSFusionDet produces more accurate detection than DINO in the illustrated light and dark validation scenes and enables effective RGB-Sonar object matching.The reported qualitative comparison describes fewer erroneous detections and omissions for RSFusionDet than DINO.
  • Limitations: Small occluded objects remain difficult because of underwater noise, and feature confusion can produce RGB-Sonar object mismatches across categories.The paper gives an example of a float in RGB being mismatched with a frustum in Sonar.
  • Visualization: CAFusion attention maps show correspondence between identical RGB and Sonar objects, indicating that RGB-object attention transfers to the corresponding Sonar object.The maps visualize attention behind CAFusion using gradients from RGB bounding-box and IoU losses.
  • Comparisons on Classes: RSFusionDet achieves the best performance on all evaluated classes, including reflector gains of +0.2/+0.5 AP and frustum gains of +0.7/+0.9 AP in RGB/Sonar.The results are presented as evidence of effectiveness across objects with different shapes.
  • Comparisons on Maximum Sonar Ranges: As maximum sonar range increases, RGB detection improvement over DINO increases while Sonar detection improvement decreases.The paper attributes this pattern to decreasing RGB feature information with distance and reduced RGB-provided information and object scale in Sonar images.

5.3. Ablation Studies

The ablations evaluate CAFusion, OMHead, and their design choices for RGB-Sonar fusion, detection, and object matching. Results support cross-modal fusion and matching, with balanced CAFusion depth performing best.

  • RSFusionDet components: CAFusion improves RGB and Sonar detection by 0.5/1.1 AP, while OMHead achieves 83.4 F1-Scorematch for cross-modal object matching.Replacing cross-attention with self-attention leaves RGB performance unchanged and improves Sonar by only 0.1 AP under the same #Params and GFLOPs.
  • RSFusionDet components: OMHead further improves detection by 0.2/0.3 AP with CAFusion, showing that object matching contributes to RGB-Sonar feature fusion.Without CAFusion, matching performance drops to 70.1 or 70.4 F1-Scorematch, indicating CAFusion’s importance for matching.
  • CAFusion design: Concatenation outperforms addition for multimodal detection and object matching because addition can discard detail and interfere across spatially inconsistent features.The passage also reports that concatenation better represents nonlinear relationships and complex feature interactions.
  • CAFusion design: Removing positional embeddings, scale-level embeddings, or Linear from CAFusion decreases RSFusionDet performance.Positional embeddings help learn spatial correspondence between RGB and Sonar object features, while scale-level embeddings provide gains for small RGB objects.
  • CAFusion design: Three CAFusion layers produce the best RGB and Sonar performance, and balancing the layer counts across modalities avoids learning imbalance.The number of layers affects RGB and Sonar differently, but unequal modality depths can slightly reduce performance.
  • OMHead design: Bounding-box priors and IoU-weighted matching loss improve object matching to 83.4 F1-Scorematch, with three OMHead encoding layers performing best.The ablation reports an 8.5 F1-Scorematch gain from box priors and a 72 F1-Scorematch improvement from IoU weighting.

6. Conclusion

The paper introduces RSFusion and RSFusionDet for underwater RGB-Sonar multimodal object detection. CAFusion and OMHead address spatially misaligned fusion and object matching, while benchmark results show gains across challenging object conditions; further research remains needed.

  • Contributions: RSFusion is introduced as an underwater RGB-Sonar multimodal object detection benchmark with evaluation metrics.The conclusion describes it as a new benchmark for this task.
  • Contributions: RSFusionDet uses CAFusion and OMHead with OMLoss to perform spatially misaligned RGB-Sonar feature fusion and object matching.CAFusion uses deformable attention for feature fusion.
  • Conclusion: RGB-Sonar multimodal detection shows adaptability and high performance improvement for dark-scene, small and medium, and distant objects on RSFusion.This conclusion is based on the experimental results reported for the RSFusion benchmark.
  • Future work: Further exploration and research are still needed for RGB-Sonar multimodal object detection beyond the RSFusion benchmark.The authors characterize their work as pioneering research and provide RSFusion as a benchmark for future study.

7. Details of RSFusion Dataset

RSFusion documents RGB and Sonar data, their spatial and feature differences, sonar conversion, object distributions, and annotations. The dataset supports analysis across modalities, scenes, ranges, and object scales.

  • Dataset overview: RSFusion visualizes all object classes in paired RGB and Sonar modalities, revealing richer RGB features and spatial feature misalignment.The object-class figure places RGB examples on the left and Sonar examples on the right.
  • Sonar processing: Sonar sector images are converted from raw images using beam angles and echo intensities from a forward-looking multibeam sonar with a 130-degree horizontal field of view.The sonar sensor returns a raw image and a list of beam angles during collection.
  • Object properties: Object scales follow COCO-style thresholds: small objects are below 32 * 32 pixels, medium objects range from 32 * 32 to 96 * 96, and large objects exceed 96 * 96.RGB objects are predominantly large, with metal polyhedron and float concentrated among medium objects.
  • Object distributions: Most RGB images contain 0, 1, or 2 objects, whereas Sonar images average 2.5 objects per image, one more than RGB images; both modalities have a maximum of 7.Some paired samples contain objects in one modality but none in the other.
  • Scene and range distributions: The number of objects in light scenes is about three times that in dark scenes, while sr15 contains more objects than sr5, sr10, and sr20.The srx notation denotes a maximum sonar range of x meters; sr5 and sr15 contain more small objects than sr10 and sr20.
  • Annotations and splits: RSFusion is divided equally into training and validation sets in an 8:2 ratio, with separate RGB and Sonar annotation files and modality-specific image statistics.RGB statistics are reported in BGR channel order, while Sonar statistics are reported for grayscale images.
Loading 2608.25367v1…