Source-linked AI summary
Beyond Pixel Similarity: Task-Aware Evaluation of GAN-Based Synthetic Sonar Data for Robotic Perception
Hannan Ejaz Keen, Muhammad Moazam Fraz, Karsten Berns
TL;DR
The paper examines whether conventional pixel-level fidelity metrics capture the usefulness of GAN-generated sonar imagery for robotic perception. It compares Pix2Pix models with four discriminator receptive-field configurations using SSIM, PSNR, MSE, and detectors trained on real sonar data. Across the considered datasets and models, the best fidelity scores do not consistently yield the best detection performance, motivating task-aware evaluation.
Problem
It is unclear whether SSIM, PSNR, and MSE adequately reflect the downstream perception usefulness of GAN-generated synthetic sonar data.
Method
Pix2Pix models with PixelGAN, PatchGAN-16, PatchGAN-70, and ImageGAN discriminators are evaluated using fidelity metrics and real-trained object detectors on two sonar datasets.
Results
Across the two datasets, the configuration with strongest image-fidelity scores does not consistently achieve strongest object-detection performance; PatchGAN variants frequently perform better despite lower similarity.
Takeaways & Limitations
Pixel-level metrics capture only part of synthetic sonar quality relevant to perception and should be complemented with task-aware evaluation.
Abstract
from arXiv · showhide
Synthetic data can reduce the cost of collecting and annotating training data for robotic perception, but generating sensor observations that preserve the characteristics relevant to downstream perception remains challenging, particularly for sonar imagery. In this work, we investigate whether conventional image-fidelity metrics adequately reflect the downstream perception performance of GAN-generated synthetic sonar data. We employ a Pix2Pix conditional generative adversarial network with four discriminator configurations characterized by different receptive fields: PixelGAN, PatchGAN-16, PatchGAN-70, and ImageGAN. The models are trained using sonar imagery from two datasets and evaluated using conventional image-fidelity metrics, including Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Mean Squared Error (MSE). To complement these pixel-level measures with task-oriented evaluation, YOLOX-S, YOLOX-L, and Faster R-CNN detectors are trained exclusively on real sonar imagery and subsequently evaluated on the GAN-generated images using identical test samples and annotations across all discriminator configurations. The results reveal a discrepancy between image-fidelity and downstream object-detection performance: the configuration achieving the best SSIM, PSNR, and MSE does not consistently yield the best detection performance. In particular, PatchGAN configurations achieve strong downstream detection results despite not achieving the highest pixel-level similarity scores. These findings suggest, for the datasets and models considered, pixel-level image-fidelity metrics alone may not consistently capture the task-relevant realism of synthetic sonar observations and motivate the use of task-aware evaluation for synthetic sensor data intended for robotic perception.
I. INTRODUCTION
Synthetic sonar generation can reduce costly real-world data collection, but realistic acoustic appearance and task-relevant perception remain difficult to reproduce. This work tests whether pixel-level fidelity metrics reflect object-detection performance across Pix2Pix discriminator scales.
- I. INTRODUCTION: Sonar imagery contains speckle noise, target highlights, acoustic shadows, reverberation, and strong variation with range, viewing angle, and seabed conditions.
- I. INTRODUCTION: Synthetic data can provide many labeled observations, but conventional simulation may model scene geometry without reproducing real sonar appearance.
- I. INTRODUCTION: Pixel-level metrics such as SSIM, PSNR, and MSE do not directly assess whether detection-relevant boundaries, shadows, and contrast patterns are preserved.
- I. INTRODUCTION: The study compares PixelGAN, PatchGAN-16, PatchGAN-70, and ImageGAN using image-fidelity metrics and detectors trained only on real sonar imagery.
- I. INTRODUCTION: The strongest pixel-level fidelity does not consistently produce the strongest detection performance, with PatchGAN configurations often performing better despite lower similarity scores.
A. Sonar Datasets and Conditioning
The study uses two sonar datasets with different acquisition conditions and converts their annotations into a shared mask-based conditioning representation. Preprocessing removes redundant video frames and retains selected augmentations that avoid artifacts.
- A. Sonar Datasets and Conditioning: Marine Debris imagery comes from a controlled pool, whereas UATD imagery comes from a more variable river environment.
- A. Sonar Datasets and Conditioning: Segmentation masks condition Marine Debris generation, while UATD bounding boxes are converted into binary object-versus-background masks.
- A. Sonar Datasets and Conditioning: Similar consecutive video frames are filtered, and mirroring plus random jitter are retained after other transformations introduce undesirable artifacts.
B. Pix2Pix-Based Sonar Image Generation
The generation system uses a fixed Pix2Pix U-Net generator while varying discriminator receptive field across four configurations. Its objective combines conditional adversarial learning with L1 reconstruction to balance target-domain appearance and paired-image correspondence.
- B. Pix2Pix-Based Sonar Image Generation: The Pix2Pix generator uses a U-Net, while training data, preprocessing, optimization, and generator architecture remain fixed across discriminator experiments.
- B. Pix2Pix-Based Sonar Image Generation: The conditional adversarial objective is combined with an L1 reconstruction loss.
- B. Pix2Pix-Based Sonar Image Generation: The adversarial term encourages target-domain consistency, while the L1 term encourages correspondence with the paired reference image.
C. Evaluation Protocol
Evaluation combines reference-image fidelity with detector performance on generated images. This protocol defines task-relevant realism through features useful for detection rather than complete physical or sensor-level realism.
- C. Evaluation Protocol: Image fidelity is measured with SSIM, PSNR, and MSE against corresponding reference images, where higher SSIM and PSNR and lower MSE indicate greater similarity.
- C. Evaluation Protocol: YOLOX-S, YOLOX-L, and Faster R-CNN are trained only on real sonar data and tested on generated images using identical samples and annotations.
- C. Evaluation Protocol: Task-level evaluation reports AP averaged over IoU thresholds from 0.50 to 0.95 together with recall.
- C. Evaluation Protocol: Task-relevant realism means the extent to which generated sonar images preserve features useful for downstream detection, without implying complete physical or sensor-level realism.
- C. Evaluation Protocol: Table I summarizes image-fidelity performance for discriminator configurations on the Marine Debris and UATD datasets.
A. Image-Fidelity Results
Image-fidelity rankings depend on the sonar dataset, and strong pixel-level scores can coincide with smoother or less detailed generated images.
- PixelGAN achieves the strongest SSIM, PSNR, and MSE results on Marine Debris, whereas ImageGAN achieves the strongest values on UATD.
- Strong pixel-level scores can correspond to smoother or blurrier images, while PatchGAN configurations preserve sharper local structures.On Marine Debris, PixelGAN appears comparatively blurred; on UATD, ImageGAN contains fewer details than PatchGAN outputs.
- These observations suggest that conventional pixel-level fidelity metrics do not necessarily reflect characteristics most relevant for downstream perception.
B. Downstream Detection Results
Downstream detector rankings differ from image-fidelity rankings: PatchGAN configurations often outperform the pixel-level leaders on generated sonar imagery.
- On Marine Debris, PatchGAN-16 and PatchGAN-70 obtain higher YOLOX-S AP than PixelGAN, while PatchGAN-16 obtains the highest YOLOX-L AP.
- On UATD, PatchGAN-70 achieves the highest YOLOX-S and YOLOX-L AP despite ImageGAN providing the strongest image-fidelity scores.
- The downstream results reveal a different ranking from the image-fidelity metrics, indicating that maximizing similarity does not necessarily maximize detector-relevant feature preservation.
- For UATD, ImageGAN records SSIM = 0.760, PSNR = 32.034 dB, and MSE = 20.800, while PatchGAN-70 reaches YOLOX-S and YOLOX-L AP values of 0.72 and 0.74.
C. Discussion: Image Fidelity Versus Task-Relevant Realism
Pixel-level metrics and detector transfer evaluate different aspects of synthetic sonar realism, producing different method rankings in these experiments.
- SSIM, PSNR, and MSE reward pixel or local-structure correspondence, even when generated images smooth or modify small target structures.
- Detector evaluation instead tests whether generated images retain cues that discriminate object locations, and PatchGAN configurations show stronger downstream performance in these settings.The proposed local-region explanation is possible but was not directly isolated experimentally.
- The preferred discriminator varies with dataset and detector, so no universally optimal configuration is identified.
- The study suggests complementing image-fidelity metrics with task-aware evaluation for synthetic sonar intended for robotic perception.
- Detector transfer provides evidence of task-relevant realism, not a complete measurement of physical sensor realism or simulator-to-real domain-gap reduction.
IV. CONCLUSION
The study finds that pixel-level image-fidelity scores do not consistently predict downstream detection quality for GAN-generated sonar imagery. It therefore supports task-aware evaluation and identifies broader sensor and dataset comparisons as future work.
- The evaluation compared four Pix2Pix discriminator configurations using SSIM, PSNR, MSE, and detectors trained exclusively on real sonar data.
- Across two sonar datasets, the configuration with the strongest image-fidelity scores did not consistently achieve the strongest object-detection performance.
- PatchGAN configurations frequently produced stronger downstream detection despite lower pixel-level similarity.
- Future work should include additional sonar sensors and datasets and compare simulator outputs, generated observations, and real sonar to quantify broader domain-gap effects.