PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Session Multimodal Underwater Mapping with Acoustic and Optical Imaging

Precious Philip-Ifabiyi, Valerio Franchi, Fausto Ferreira, Nuno Gracias

arXiv:2609.17929v1cs.RO

TL;DR

Navigation drift and georeferencing uncertainties make it difficult to align sonar and optical maps collected in separate sessions. The paper jointly optimizes trajectories, landmarks, session transformations, and sensor extrinsics; camera measurement errors fall more strongly than sonar errors within multimodal tracks.

  • Problem

    Dead-reckoning drift and session-specific georeferencing uncertainties can spatially misalign acoustic and optical imagery, hindering data fusion and long-term monitoring.

    Source: §1 · p. 1

  • Method

    A factor-graph framework jointly optimizes vehicle trajectories, 3D landmarks, session-to-world transformations, and sensor extrinsics using global rigid alignment and local trajectory deformations.

    Source: §5 · p. 18

  • Results

    Camera measurement errors decrease more than sonar measurement errors within multimodal tracks, consistent with the higher uncertainty assigned to sonar measurements.

    Source: §4.2.2 · p. 12

  • Takeaways & Limitations

    Local pose adjustments substantially reduce nonlinear misalignments in cases where a purely rigid transformation is insufficient, though they do not eliminate all non-rigid effects.

    Source: §4.2.3 · p. 13

    The approach is sensitive to the optimizer’s initial guess because of the non-convex cost landscape, and automated feature matching remains future work.

    Source: §5 · p. 18

Abstract

from arXiv · show

Accurate seafloor mapping is essential for marine science, archaeology, and environmental monitoring. However, integrating data from different sensors, such as side-scan sonar and optical cameras, collected across separate survey sessions, remains challenging due to positioning drift and sensor offsets. This paper presents a multi-session, multimodal underwater mapping framework based on factor graph optimization. The method jointly optimizes vehicle trajectories, 3D landmark positions, sensor extrinsics, and per-session global alignment transformations. By combining rigid inter-session corrections with local trajectory deformations, it compensates for both inter-session offsets and intra-session distortions from accumulated navigation errors. The proposed methodology was validated on real-world datasets collected along the Catalan coast. Results show measurable improvements in map consistency over both unoptimized and rigid-alignment baselines across all metrics, including Pixel Accuracy and mean Intersection over Union. The method achieves a 3.4% improvement in pixel accuracy over the unoptimized baseline, corresponding to improved semantic labelling across approximately 14700 $\text{m}^2$ of mapped area. Qualitative results further show consistent co-registration between sonar and optical maps, even in the presence of significant trajectory distortions and inter-session misalignments. These findings demonstrate the potential of the proposed framework to generate coherent multimodal seafloor maps from heterogeneous underwater surveys.

1. Introduction

Fusing sonar and optical maps across survey sessions is hindered by navigation drift and georeferencing misalignment. The paper introduces a factor-graph framework for aligning the modalities across sessions, with feature matching and joint optimization.

1. Introduction

  • Navigation drift and separate-session georeferencing uncertainties misalign acoustic and optical imagery, complicating data fusion, interpretation, and long-term monitoring.Side-scan sonar interpretation is also labor-intensive and affected by low resolution, survey-to-survey appearance variability, geometric distortion, and acoustic noise.Source: §1 · p. 1
  • Feature matches across optical and side-scan sonar images feed a factor-graph backend that estimates globally aligned session trajectories.The framework can also align multi-session datasets consisting solely of side-scan sonar or optical imagery.Source: §1 · pp. 1–2
  • The framework globally aligns side-scan and optical images collected across different sessions into a consistent, georeferenced multimodal map.The authors describe it as the first approach to systematically address spatial alignment and fusion of these modalities across multiple survey sessions.Source: §1 · pp. 1–2
  • The framework supports datasets with filtered vehicle poses alone or with additional sensor measurements such as DVL.A constant-velocity motion model and keyframe selection are proposed to promote trajectory smoothness and computational tractability.Source: §1 · p. 2

2. Related Work

Prior work covers acoustic SLAM, cross-modal feature matching, and underwater sensor fusion, but the reviewed literature leaves side-scan sonar–camera joint mapping largely unexplored. The modalities’ operating constraints and non-overlapping fields of view complicate conventional feature-based fusion.

2.1. Acoustic Mapping

  • Side-scan sonar SLAM has used feature-based filtering and pose-graph methods as well as dense subframe optimization, while forward-looking sonar has also been studied for feature-based SLAM.The reviewed side-scan work reports reduced localization drift from sonar-image constraints.Source: §2.1 · p. 2

2.2. Multimodal Feature Matching

  • Cross-modal matching approaches transform images into a common style, match features directly, or project features between modalities using sensor parameters.The review notes limited evaluation settings for some methods and dependence on calibration and projection models for projection-based matching.Source: §2.2 · pp. 2–3

2.3. Multi-Sensor Fusion

  • The reviewed multimodal SLAM literature focuses predominantly on forward-looking-sonar–camera systems, leaving side-scan-sonar–camera joint mapping largely unexplored.Different operating altitudes and non-overlapping fields of view can prevent simultaneous observation of common scene features, obstructing standard feature-based association and fusion.Source: §2.3 · p. 3

3.1. System Overview and Data Acquisition

The system processes separate survey sessions, potentially from different platforms, and turns within- and cross-session feature matches into a unified multimodal map. Its data collection combines side-scan sonar towfish surveys with AUV optical imaging.

3.1. System Overview and Data Acquisition

  • Each session supplies sensor data and initial pose estimates; sonar-sonar, optical-optical, and sonar-optical matches are organized into tracks before optimization produces vehicle poses, landmarks, and a unified 2D map.A feature track represents observations of the same seafloor 3D point, whether acoustic, optical, or mixed.Source: §3.1 · p. 4
  • The datasets came from shallow-water seagrass, sand, and hard-bottom areas surveyed in multiple sessions using a sonar-equipped towfish and an optical-camera AUV.The towfish navigation combined vessel RTK GNSS, USBL positioning, and attitude measurements; the AUV used an INS aided by DVL and pressure sensing, with surface GNSS initialization.Source: §3.1 · p. 4
  • The AUV cameras captured high-resolution seafloor imagery, but their manually measured camera-to-body extrinsics were uncertain by about 5° in orientation and 10 cm in translation.The AUV’s navigation solution was provided by an onboard INS aided by DVL and a pressure sensor for depth.Source: §3.1 · p. 4

3.2. Feature Detection, Matching, and Tracking

The method builds optimization constraints from within-modality and cross-modal feature associations, grouping observations into tracks and filtering optical-only tracks for reliability and coverage. Optical matching uses SIFT with RANSAC outlier removal, while sonar and cross-modal correspondences are manually established.

3.2. Feature Detection, Matching, and Tracking

  • Robust data associations provide features within each sensor stream and between sensor types, forming a fundamental component of the framework.The method uses optical-optical, sonar-sonar, and sonar-optical matches as constraints in global optimization.Source: §3.2 · p. 4

3.2.1. Optical-Optical Feature Matching

  • SIFT detects and describes features between consecutive optical images, and RANSAC removes outlier matches used to establish local motion constraints.No explicit image preprocessing was performed for the adequately illuminated near-seafloor data.Source: §3.2.1 · pp. 4–5

3.2.2. Sonar-Sonar and Sonar-Optical Feature Matching

  • Sonar-sonar and sonar-optical correspondences are established manually because robust automated matching remains challenging across modalities and in high-texture, low-feature regions such as seagrass.Annotations target recognizable landmarks and topographical edges shared by sonar waterfall displays and optical frames.Source: §3.2.2 · p. 5

3.2.3. Feature Tracking and Selection

  • Feature tracks group observations of the same 3D landmark over time and may contain one modality or a mixture of sonar and optical observations.Tracks provide the constraints for global optimization.Source: §3.2.3 · p. 5

3.2.4. Optical-Only Feature Track Filtering Algorithm

  • The optical-only track filter unions each image’s N longest tracks, prioritizing persistent features distributed across the sequence and discarding short, potentially spurious tracks.The method recommends setting N to at least 4 tracks to improve trajectory reconstruction.Source: §3.2.4 · p. 5

3.3. Multi-Session Optimization

This section describes the multi-session factor-graph optimization formulation and the initialization of its session states, anchors, camera extrinsics, and landmarks. It covers modality-specific observations and navigation and motion constraints, with sparse key states and track-dependent landmark initialization.

3.3.1. Problem Formulation

  • The factor graph jointly optimizes vehicle trajectories, landmark positions, and sensor extrinsics, and supports flexible fusion of heterogeneous information sources; vehicle states are 9-DOF navigation states.Each session anchor represents its pose offset with respect to the global coordinate frame and defines the session’s local reference frame.Source: §3.3.1 · p. 5

3.3.2. Data Pre-processing

  • Sonar keypoints identified by ping, side, and bin are converted to metric slant range by linear interpolation; their along-track measurement is set to zero.The zero along-track distance represents an assumed observation on a line perpendicular to vehicle travel.Source: §3.3.2 · p. 5

3.3.3. Factor Definitions and Error Models

  • Navigation factors distinguish drifting horizontal position from bounded depth and orientation, while motion, conditional DVL, initial-state, anchor, and camera-extrinsic priors constrain the graph.The method uses relative motion constraints for horizontal states and unary absolute-measurement factors for depth and orientation.Source: §3.3.3 · pp. 6–7
  • Camera and sonar factors project shared 3D landmarks into image or slant-range/along-track measurements, with sonar measurement covariance adapted to feature uncertainty and beam width.The sonar model maps waterfall-image localization uncertainty into metric range and along-track variance.Source: §3.3.3 · pp. 7–8

3.3.4. Keyframe Selection

  • To avoid prohibitively large graphs, key states include every state with a landmark observation plus states inserted at fixed time intervals to represent trajectory shape and smoothness.This sparse selection is motivated by dense trajectories that can contain tens of thousands of poses.Source: §3.3.4 · pp. 8–9

3.3.5. Non-Linear Least-Squares Cost Function

  • Assuming Gaussian noise, MAP estimation becomes nonlinear least squares over squared Mahalanobis factor errors; camera projection factors use Huber loss to limit the effect of matching outliers.The problem is solved with GTSAM’s Levenberg–Marquardt algorithm.Source: §3.3.5 · p. 9

3.3.6. Initial Estimates

  • The optimization requires initial estimates for every graph variable, with good initial guesses intended to improve convergence speed and reliability.The supplied passage introduces initialization strategies but does not specify them.Source: §3.3.6 · p. 9
  • Each session starts at identity, derives relative poses from raw navigation and velocities by finite differencing, anchors at its first georeferenced pose under a loose prior, and initializes camera extrinsics under a tight prior.Source: §3.3.6 · p. 9
  • Sonar-only landmarks use the centroid of measurements projected onto a seafloor plane, while camera-only landmarks use standard triangulation.Source: §3.3.6 · pp. 9–10
  • Multimodal landmarks use observation-count-weighted sonar and camera estimates; single-view or failed-triangulation cases fall back to sonar estimates or altitude-plane ray intersections.For a single camera view in a multimodal track, the sonar estimate is used; when camera-only triangulation fails because of a chirality violation, the image ray is intersected with a horizontal plane at the AUV's capture-time altitude.Source: §3.3.6 · p. 10

3.4. Post-Processing and Map Generation

After optimization, sparse key-state poses are interpolated into dense trajectories, which are then used to generate georeferenced maps for each sensor modality.

3.4. Post-Processing and Map Generation

  • Optimization yields sparse poses at key states, so intermediate poses are estimated to produce a continuous trajectory.This post-processing follows convergence of the optimization.Source: §3.4 · p. 10
  • Dense trajectories are interpolated between sparse key-state poses along the SE(3) geodesic, representing shortest constant-velocity motion between poses.The relative transformation is mapped to the Lie algebra, scaled for the target timestamp, and mapped back to SE(3).Source: §3.4 · p. 10
  • The final dense, globally aligned trajectories generate georeferenced maps for optical imagery and side-scan sonar.MosaicViewer produces the optical map, while MB-System produces a georeferenced GeoTIFF of acoustic backscatter.Source: §3.4 · p. 10

4. Experimental Results & Discussion

Experiments on sonar and optical surveys from the same geographical area evaluate trajectory alignment, reprojection errors, and multimodal map consistency. Optimization combines session-level alignment with local trajectory changes, improving map alignment over unoptimized and rigid baselines while leaving some non-rigid effects unresolved.

4.1.1. Dataset Description

  • Eight sessions—four sonar and four optical—collected in the same geographical area provide the evaluation dataset.The sonar was collected using a Klein System 3000 side-scan sonar; optical data came from Girona 1000 AUV down-looking cameras.Source: §4.1.1 · p. 10

4.1.2. Feature Matching

  • Optical-optical matches substantially outnumbered other match types, while cross-modal matches were comparatively sparse and manually identified.Cross-modal correspondences were selected at seabed contours and distinctive landmarks.Source: §4.1.2 · p. 11

4.1.3. Implementation Details

  • The optical-track selection parameter N is 4, and keyframes are selected at fixed one-second intervals.Other unspecified parameters are referred to the source code.Source: §4.1.3 · p. 11

4.2. Evaluation

  • Evaluation combines internal consistency, alignment accuracy of segmented maps, qualitative inspection, and comparison against rigid inter-session transformations.Ground-truth trajectories and multimodal maps are unavailable.Source: §4.2 · p. 11

4.2.1. Session Trajectories

  • Optimization globally transforms session anchors and locally deforms trajectories, with mostly smooth deformations particularly visible in sonar session SON2.The smooth changes are consistent with the constant-velocity factor accounting for vehicle dynamics.Source: §4.2.1 · p. 11

4.2.2. Reprojection Error

  • Reprojection-error medians and spreads decrease across track categories except for sonar measurements within multimodal tracks; camera errors fall more strongly than sonar errors.Camera-only outliers are attributed mainly to inaccurate feature matches, with the Huber loss limiting their influence; sonar’s higher modeled uncertainty gives camera measurements more weight.Source: §4.2.2 · pp. 11–12

4.2.3. Multimodal map

  • Before-and-after mosaic overlays show substantially improved co-registration, while residual non-rigid effects remain and further error modeling and factor-weight tuning are left for future work.The method uses both global rigid session corrections and local trajectory deformation.Source: §4.2.3 · pp. 12–13

4.2.4. Comparison with Rigid Baseline

  • The proposed method performs slightly better than rigid optimization across all map-alignment metrics, and its 3.4% pixel-accuracy improvement corresponds to approximately 14,700 m² of mapped area.Both rigid optimization and the proposed method improve alignment scores over unoptimized maps.Source: Table 6 · p. 15, §4.2.4 · p. 15

4.3. Trajectory Smoothness

The constant-velocity approach produces smoother trajectories than relative-pose factors across sonar and optical sessions. Increasing relative-pose factor noise weakens pose links and increases trajectory jerk.

4.3. Trajectory Smoothness

  • Relative-pose factors were tested at small, medium, and high noise levels, with other factor noise models held identical across runs.The between-pose noise standard deviations are defined in the tangent space of SE(3), with orientation in radians and translation in meters.Source: §4.3 · p. 16, Table 7 · p. 17
  • The constant-velocity approach yields lower jerk than relative-pose factors across all sonar and optical sessions.Figure 13 compares NARJ in the x and y directions; NARJ measures the third time derivative of position.Source: §4.3 · p. 17, Figure 13 · p. 17
  • Higher relative-pose factor noise weakens links between poses, allowing individual poses to move more independently and increasing trajectory jerk.Lower noise links neighboring poses more rigidly, so nearby poses move together during corrections.Source: §4.3 · p. 17
  • Because relative-pose measurements are pseudo-measurements providing no new information, the authors favor constant-velocity factors to constrain consecutive poses.The paper reports that this choice ensures trajectory smoothness.Source: §4.3 · pp. 16–17

4.4. Convergence Test for Anchors

Anchor estimates become more sensitive to initialization as perturbation noise increases, with the solver often reaching different local minima at higher noise. The paper attributes this sensitivity to a non-convex cost landscape and notes that sufficiently accurate GNSS initialization places anchors within the global optimum's basin of attraction.

4.4. Convergence Test for Anchors

  • Anchor translation error and its variability increase as initialization noise grows, and the solver increasingly converges to different local minima.Figure 14 reports mean 3D translation error and standard deviation across perturbations relative to noise-free initialization.Source: §4.4 · p. 17, Figure 14 · p. 18
  • Combining range, plane, and camera-projection factors induces a non-convex cost landscape, while sufficiently accurate GNSS initialization places anchors within the global optimum’s basin of attraction.The paper identifies extending certifiably correct solver guarantees to heterogeneous multimodal factor graphs as an open problem.Source: §4.4 · pp. 17–18

5. Conclusion

The paper concludes that its factor-graph framework aligns side-scan sonar and optical data across sessions through global rigid transformations and local trajectory deformations. It reports reduced projection errors and a 3.4% pixel-accuracy improvement, while noting sensitivity to the optimizer’s initial guess.

5. Conclusion

  • The framework jointly optimizes trajectories, 3D landmarks, session-to-world transformations, and sensor extrinsics, combining per-session rigid alignment with local trajectory deformations.The paper reports that this two-level alignment corrects inter-session offsets and intra-session distortions caused by navigation drift.Source: §5 · p. 18
  • The method reduced projection errors across sensor modalities and improved multi-session multimodal map pixel accuracy by 3.4%.Qualitative results showed correction of significant initial misalignments and coherent sonar-optical coregistration.Source: §5 · p. 18
  • The approach remains sensitive to the optimizer’s initial guess because of the non-convex cost landscape; global optimality under arbitrary initialization is future work.The authors also identify automated feature matching as a direction for future work.Source: §5 · p. 18

CRediT authorship contribution statement

The statement assigns conceptualization and methodology to all four authors, with distinct software, drafting, supervision, review, and data-collection roles.

CRediT authorship contribution statement

  • Precious Philip-Ifabiyi is credited with software and the original draft; Valerio Franchi and Fausto Ferreira with supervision and review; Nuno Gracias with data collection and supervision.All four authors are credited with conceptualization and methodology.Source: CRediT authorship contribution statement · p. 18
Loading 2609.17929v1…