PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Vision Foundation Models to Acoustics for Pose-Free 3D Sonar Reconstruction

Kevin Zhang, Jingxi Chen, Mohamad Qadri, Russell Shomberg, Michael Kaess, Jia-Bin Huang, Adithya Pediredla, Christopher Metzler

arXiv:2609.06261v1cs.CV

TL;DR

Prior sonar 3D reconstruction methods require accurate poses, which are difficult to obtain underwater. This paper adapts a multiview diffusion model to sonar using geometric correspondence and demonstrates pose-free 3D reconstruction from sonar video.

  • Problem

    Prior sonar-based 3D reconstruction methods require accurate sonar poses, which are difficult to obtain underwater.

    Source: §I · p. 2

  • Method

    The method uses geometric correspondence to train a multiview diffusion model to translate sonar measurements into grayscale images, which a feedforward model uses for 3D reconstruction.

    Source: §I · p. 2, A. Imaging Sonar and Perspective… · p. 3

  • Results

    The paper experimentally demonstrates pose-free 3D reconstruction from sonar video.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    The pipeline reconstructs scenes from unposed sonar images while recovering camera poses via VGGT.

    Source: B. Simulation Results · p. 6

Abstract

from arXiv · show

Vision foundation models trained on Internet-scale RGB datasets enable remarkable capabilities across a range of tasks, from text-to-video generation to few-shot 3D scene reconstruction. An acoustic foundation model trained on large-scale sonar datasets could enable similar capabilities in the underwater domain, where turbidity and low-visibility conditions make conventional RGB foundation models inapplicable. Unfortunately, a lack of freely available large-scale sonar datasets makes training such a model from scratch impractical. In this work, we demonstrate that vision foundation models can be efficiently adapted to the sonar setting by (1) exploiting the geometric relationship between the two sensing modalities and (2) employing accurate physics-based noise models for synthetic data generation. The resulting sonar adaptation models enable new capabilities: For the first time, we experimentally demonstrate sonar-based pose-free 3D reconstruction.

I. INTRODUCTION

The paper adapts vision foundation models to imaging sonar for pose-free 3D reconstruction. It uses geometric sonar–camera correspondence and synthetic paired data to translate sonar video into grayscale views for reconstruction.

I. INTRODUCTION

  • Prior sonar-based underwater reconstruction methods require accurate poses, which are difficult to obtain because GPS is unavailable, inertial odometry drifts, and sonar SLAM can be degeneracy-prone.Source: §I · p. 2
  • When objects are sufficiently far from the sensors relative to their extent, forward-looking sonar measurements closely match those from an orthogonally positioned perspective camera.Source: §I · p. 2
  • The method fine-tunes a multiview diffusion model to synthesize grayscale views from sonar video, then feeds those views to a feedforward 3D reconstruction model.Source: §I · p. 2
  • The contributions include a sonar–grayscale pairing method with a quantitative bound on projected-coordinate differences and a diffusion model trained on synthetic paired data.Source: §I · p. 2
  • The pipeline demonstrates pose-free underwater 3D reconstruction from real imaging sonar data.The authors describe this as the first such demonstration.Source: §I · p. 2

II. RELATED WORK

Related work spans feedforward 3D vision models, video diffusion, and sonar reconstruction. The paper distinguishes its approach by adapting pretrained visual models to recover sonar geometry without pose information.

II. RELATED WORK

  • Feedforward 3D vision methods estimate scene structure from camera images, including point-map approaches beginning with Dust3R and later multi-view extensions [1],.Source: §II · p. 2
  • Video diffusion has been used for novel-view synthesis and downstream 3D reconstruction, while SV3D generates novel views from object-centric inputs and motivates this paper’s approach.Source: §II · p. 2
  • Sonar reconstruction methods address missing elevation with supervised learning,,, optimization,, graph processing, and ICP-based methods.Source: §II · p. 2
  • Differentiable-rendering approaches recover sonar geometry, while pose estimation remains challenging because reliable feature correspondence is difficult; some methods require coarse pose initialization.Source: §II · p. 2
  • Prior sonar diffusion work focused on simple simulated geometries or pose-dependent multibeam data, whereas this method targets pose-free reconstruction from 2D imaging sonar.Source: §II · pp. 2–3

III. METHOD

The method fine-tunes an image-to-video model to translate sonar video into grayscale video, then uses the synthesized views for 3D reconstruction. Training uses paired sonar and grayscale videos rendered from synthetic scenes.

III. METHOD

  • Figure 2 contrasts sonar and perspective-camera projection models and shows a sensor arrangement with its resulting sonar and grayscale images.Source: Figure 2 · p. 3
  • The pipeline fine-tunes SV3D to convert imaging-sonar video to grayscale video, then passes the synthesized images to VGGT for pose-free 3D reconstruction.Source: §III · p. 3
  • Training pairs are rendered from synthetic scenes: sonar video is the input, and visually similar grayscale video is the target.Source: §III · p. 3

A. Imaging Sonar and Perspective Camera Correspondence

The section constructs paired sonar and grayscale perspective views by matching their weak-perspective projections. The correspondence is approximate under stated camera-distance and point-elevation conditions.

A. Imaging Sonar and Perspective Camera Correspondence

  • The method uses matching weak-perspective approximations of sonar and perspective-camera projections to construct corresponding grayscale training images and bound their relative error.This correspondence is used to generate simulation-based training data.Source: A. Imaging Sonar and Perspective… · pp. 3–4
  • Imaging sonar measures range and azimuth but collapses elevation, while optical cameras recover image coordinates but not depth.The section seeks a weak-perspective approximation under which the two image-formation models yield similar observations.Source: A. Imaging Sonar and Perspective… · p. 3
  • The corresponding camera is placed with its optical axis parallel to the sonar projection axis and orthogonal to the sonar viewing direction.Its image plane is parallel to the sonar xy-plane, so projections onto the camera plane and virtual plane differ by uniform scaling and translation.Source: A. Imaging Sonar and Perspective… · p. 3
  • When |Z|/Zc ≪ 1, the perspective-camera projection approaches the perpendicular projection onto the sonar xy-plane.This condition requires scene extent along the camera optical axis to be small relative to camera distance.Source: A. Imaging Sonar and Perspective… · pp. 3–4
  • The sonar projection approximation additionally requires |Z|/r ≪ 1, meaning elevation is small relative to planar range.The section combines projection differences with the triangle inequality to bound the relative difference between perspective and imaging-sonar projections.Source: A. Imaging Sonar and Perspective… · p. 4

B. Physics-informed Noise Model

The synthetic sonar data are corrupted with a physics-informed noise model to better reproduce real-sonar degradation patterns. It models range diffusion, speckle, range-dependent additive noise, azimuthal streaks, and directional blur.

B. Physics-informed Noise Model

  • The synthetic data receive physics-informed corruption intended to reproduce degradation patterns observed in real sonar data.Realistic noise modeling is used because the model must generalize from synthetic training data to real sonar measurements.Source: B. Physics-informed Noise Model · p. 4
  • Range diffusion models unresolved scatterers in a finite-resolution cell, biasing measured range outward from the nearest surface and diminishing intensity.The model follows HoloOcean and weights the kth return intensity using an exponential-distribution density before range and azimuth binning.Source: B. Physics-informed Noise Model · p. 4
  • Speckle is modeled by spatially filtering circular complex Gaussian noise, introducing local intensity correlations while preserving pointwise distributions.Larger σ produces smaller speckle; the spatial filtering is the primary difference from HoloOcean’s speckle model.Source: B. Physics-informed Noise Model · p. 4
  • The model adds noise whose magnitude increases with range, with the noisy image formed by adding it to the speckled image.The additive-noise description follows HoloOcean.Source: B. Physics-informed Noise Model · p. 4
  • Azimuthal streaks are simulated by boosting bins at a randomly selected small fraction of ranges, followed by Gaussian blurring along radial and azimuthal directions.Streaks represent high intensities across azimuth angles caused by strong returns at a given range.Source: B. Physics-informed Noise Model · p. 4

C. Model Architecture

The sequence-to-sequence model adapts SV3D to map sonar sequences into grayscale frames, then uses VGGT to recover a 3D point cloud.

C. Model Architecture

A. Simulation Setup

The setup renders grayscale targets and sonar inputs from synthetic objects, then trains on paired views with perturbed poses and randomized sonar noise.

A. Simulation Setup

  • Training and testing use renders of 900 and 100 Objaverse-XL objects, respectively.Grayscale targets come from Blender; a custom renderer implementing the HoloOcean forward model generates sonar images.Source: A. Simulation Setup · p. 5
  • The grayscale targets emphasize geometry rather than material appearance because sonar imagery is monochrome and primarily encodes geometry.Lighting and materials are adjusted for geometric contrast and relatively uniform gray color.Source: A. Simulation Setup · p. 5

ORTHOGONAL, CROPPED CARTESIAN PAIRING ACHIEVES THE BEST

B. Simulation Results

Simulation results show that reconstruction quality depends on sonar-camera pairing and vertical field of view, while synthesized views can recover missing structure.

a) Ablating sonar-image pairing strategies

  • Cropped Cartesian sonar with orthogonal sonar-camera placement achieves the best PSNR, SSIM, and LPIPS among the tested pairings.The study compares sonar image domains and sensor orientations.Source: Figure 4 · p. 5, B. Simulation Results · p. 5

b) Qualitative results

  • Across diverse test geometries, synthesized views restore missing structural details across frames and remove degradations present in sonar inputs.Examples include an axe head and a basket handle absent from corresponding input frames.Source: B. Simulation Results · p. 5, Figure 5 · p. 6
  • Repeating one view 21 times instead of using multiview inputs lowers PSNR by 4.99 dB and SSIM by 0.067, while increasing LPIPS by 0.179.Source: B. Simulation Results · p. 6

c) 3D reconstruction comparisons

  • As VFOV increases, our method’s Chamfer distance generally decreases while the sonar-based baselines degrade, even though they receive 84 views versus our 21.The authors attribute the difference to greater object coverage and closer resemblance to grayscale images at wider VFOV.Source: B. Simulation Results · p. 6
  • VGGT recovers poses accurately in the higher-VFOV regime despite receiving unposed sonar images at inference.Predicted poses are evaluated against ground truth after Sim(3) alignment using rotational and translational errors.Source: B. Simulation Results · p. 6

C. Hardware Experiments

Real sonar experiments compare raw measurements with method-processed outputs, while the supplied passages also describe pose-recovery and VFOV observations.

Real-data setup

  • The real-data setup suspends an object from a float by ropes and spins the float in front of the imaging sonar.The sonar is mounted on an acrylic support secured to the tank railing.Source: C. Hardware Experiments · p. 6

Pose recovery

  • VGGT successfully recovers poses when elevation exceeds 10 degrees; the simulation cameras are about 1.25 meters from the origin.Source: Figure 6 · p. 6

VFOV comparison

  • At the same pose, the wide-VFOV sonar measurement visually resembles the grayscale reference capture, unlike the narrow-VFOV measurement.Source: Figure 7 · p. 6

Real-data reconstruction results

  • Across all four real-data objects, the method reduces Chamfer distance relative to VGGT applied directly to raw sonar.Distances change from 1.5 cm to 0.66 cm for the anchor, 1.3 cm to 0.96 cm for the four-way, 2.3 cm to 1.3 cm for the handlebar, and 0.91 cm to 0.47 cm for the propeller.Source: C. Hardware Experiments · p. 6

V. CONCLUSION

The paper presents a pose-free 3D reconstruction pipeline for imaging sonar, translating sonar measurements into synthesized grayscale images for VGGT. It reports real-data reconstruction while noting that the method relies on sufficient sonar evidence and object-centric scenes.

V. CONCLUSION

  • The pipeline translates sonar measurements into synthesized grayscale images, which VGGT uses to recover a 3D point cloud without sonar poses.The pipeline finetunes a video diffusion model using synthetic sonar-grayscale pairs generated from geometric correspondence between sonar and perspective-camera projections.Source: §V · p. 6
  • Across VFOVs from 1° to 40°, average Chamfer distance results show our method improves with elevation and outperforms NeuSIS and SonarSplat at larger elevation values.Source: Figure 8 · p. 7
  • Real-object reconstructions are dimension-calibrated; Chamfer distance is reported in centimeters, and precision, recall, and F1 use τ = 2% of the calibrated ground-truth bounding-box diagonal.Source: Table II · p. 7, §V · p. 7
  • Training without noise produces jagged outputs, HoloOcean noise produces blurry reconstructions, and blurred speckle produces cleaner outputs.Source: Figure 11 · p. 7
  • On real sonar data, the method enables pose-free reconstruction; Fig. 9 juxtaposes raw sonar, synthesized grayscale images, VGGT reconstructions, and ground truth.Source: §V · p. 7, Figure 9 · p. 7
  • The method relies on a generative prior that may produce incorrect geometry when sonar evidence is insufficient, and assumes object-centric scenes satisfying the stated geometric correspondence.Source: §V · p. 7

Related papers

Loading 2609.06261v1…