PaperlayerView exact revision on arXiv

Source-linked AI summary

BenthicFlow: Generating Extensible Underwater Environments via Flow Matching

Joaquín Figueira, Camile Lendering, Manfred Gonzalez-Hernandez, Giacomo D'Amicantonio, Erkut Akdag, Egor Bondarev

arXiv:2608.23173v1cs.CV

TL;DR

Underwater scene understanding is challenged by limited 3D supervision and poor generalization from surface-trained models. BenthicFlow uses one conditional flow-matching model to generate aligned RGB-depth mosaics and lift them into 3D environments. Across reefs and preprocessing variants, it closely matches test data distributions, with FID ≤20.

  • Problem

    Underwater scene understanding lacks reliable camera poses and dense 3D supervision for collected imagery.

    Source: §1 · p. 1

  • Method

    BenthicFlow uses one conditional flow-matching model to generate aligned RGB-depth mosaics, then lifts them into 3D environments with surface-aligned Gaussian surfels.

    Source: §1 · p. 2, §6 · p. 15

  • Results

    FID ≤20 across all reefs and preprocessing variants, with BenthicFlow closely matching the test data distribution.

    Source: §5.2 · p. 10

  • Takeaways & Limitations

    The framework provides a scalable basis for generating synthetic underwater environments for downstream perception and robotics applications.

    Source: §6 · p. 15

    The pipeline remains fundamentally 2.5D and cannot recover geometry absent from the generated view.

    Source: §6 · p. 15

Abstract

from arXiv · show

Computer vision applications for 3D scene understanding in underwater environments remain challenging due to the lack of high-quality 3D data and the inability of surface-trained models to generalize to underwater scenes. To address this challenge, an emerging trend is to employ generative models to close the data domain gap. However, existing methods assemble large scenes by stitching independently generated tiles post hoc with separately trained models, while demonstrating heterogeneous landscapes only within individual survey sites. We introduce BenthicFlow, a unified framework based on a single conditional flow-matching model that jointly generates aligned textures and depth maps. A MultiDiffusion-inspired sampling procedure reconciles overlapping windows throughout the generative trajectory, enabling spatially extensible RGBD mosaics without a separate stitching model. The generated mosaics are subsequently lifted into explicit 3D benthic environments using surface-aligned Gaussian surfels. Experiments across geographically distinct survey sites demonstrate that BenthicFlow preserves site-specific appearance while generating coherent, large-scale 3D scenes that closely match the target distributions. Code and trained models are available at https://github.com/jacomof/BenthicFlow.

1 Introduction

BenthicFlow addresses the scarcity of usable underwater 3D data with a single flow-matching pipeline that generates extensible RGBD mosaics and lifts them into continuous 3D scenes.

1 Introduction

  • AUV surveys provide abundant imagery, but missing captions, reliable poses, and dense 3D supervision, alongside failures of SfM, SLAM, and active depth sensing, leave usable 3D data scarce.Survey-trained generation is motivated by marine simulation, ecological anomaly detection, and the mismatch between survey frames and general-purpose text-to-image outputs.Source: §1 · pp. 1–2
  • The closest prior generator [52] assembles independently sampled tiles post hoc, requiring separate in-painting and enforcing coherence only between neighboring tile pairs within one survey site.Its seam regions lack the latent representations guiding the rest of the scene, and geographically distinct sites were not demonstrated.Source: §1 · p. 2
  • BenthicFlow trains one conditional flow-matching model to synthesize aligned RGB-depth pairs by combining flow matching with monocular depth estimation.The pipeline leverages marine imagery from the Squidle+ collaborative data framework.Source: §1 · p. 2
  • Generated RGBD mosaics are projected into point clouds and reconstructed as spatially continuous 3D benthic environments using surface-aligned Gaussian splats.Source: §1 · pp. 2–3
  • Overlapping windows are reconciled by averaging predicted velocities throughout a shared sampling trajectory, enabling extensible mosaics without a separate stitching or in-painting network.Generation begins from reference seeds distributed across a plane, and all canvas regions share latent conditioning.Source: §1 · pp. 2–3
  • A single conditional velocity field spans geographically distinct survey sites and interpolates between them, extending generation beyond the within-site transitions shown by prior underwater generators.The introduction describes this cross-site capability as not demonstrated by prior underwater generators.Source: §1 · p. 3

2 Related Works · 3 Dataset

Related work frames BenthicFlow as a learned approach to extensible underwater scene generation, building on continuous flow matching and spatial blending. Its dataset comprises more than 600,000 benthic frames, with estimated depth and two preprocessing configurations.

2.1 Marine Computer Vision

  • Underwater vision spans marine biology, infrastructure inspection, AUV navigation, mapping, and archaeology, while optical degradation motivates restoration and scattering-aware 3D reconstruction.Examples include SeaThru, Semi-UIR, DPF-Net, and SeaThru-NeRF [24], with later work addressing dynamic scenes and processing efficiency [25].Source: §2.1 · p. 3
  • Synthetic underwater data often serves downstream tasks: transfer models learn optical characteristics, while UW-GAN generates data to derive depth estimates.The related method [52] also jointly generates images and depth using differentiable rendering and spatial fields of foundation-model latents.Source: §2.1 · p. 3
  • Unlike [52], BenthicFlow reconciles overlapping windows during generation rather than using a separately trained inpainting model after independent tile sampling.Procedural environments use deterministic assets but struggle to reproduce organic variation and complex marine light scattering [52].Source: §2.1 · pp. 3–4

2.2 Continuous Generative Modeling and Latent Representations

  • Conditional Flow Matching regresses a velocity field along per-sample conditional paths, enabling near-straight trajectories and competitive sample quality with fewer function evaluations than denoising diffusion models.Flow Matching has been scaled for image synthesis in systems including Qwen-Image [46], FLUX.2, and rectified flow transformers.Source: §2.2 · p. 4
  • BenthicFlow extends Representation Autoencoders to joint RGBD tokens, building on semantic spaces from frozen vision foundation models to ease Transformer learning dynamics.The passage contrasts RAEs with VQ-GANs and VAEs, whose information bottlenecks can yield highly nonlinear distributions that complicate velocity estimation.Source: §2.2 · p. 4
  • BenthicFlow extends spatial blending into continuous flow trajectories by combining overlapping windows directly within velocity-estimation steps.MultiDiffusion averages overlapping crops during sampling; concurrent work [4] uses Gaussian-weighted averaging that prioritizes window centers.Source: §2.2 · p. 4

3 Dataset

  • The dataset contains over 600,000 color-normalized, 518 × 518 benthic nadir frames from two Australian reefs and Hawaii’s northwestern coast.Frames span marine fauna, flora, substrates, and anthropogenic debris, and are organized by campaigns and sequential deployments.Source: §3 · p. 4
  • Splits are made at the deployment level to prevent temporal leakage, reserving approximately 90,000 images each for validation and testing and 500,000 for training.Because standard surveys lack dense geometry, the dataset retains estimated per-pixel depth as a fourth channel; BenthicFlow uses CLAHE and Depth Anything V2, whereas BenthicFlow-DPF uses DPF-Net for enhancement and depth estimation.Source: §3 · p. 5

4 Methodology

BenthicFlow generates large underwater RGBD mosaics by conditioning latent-space flow matching on reference imagery, reconciling overlapping windows during sampling, and lifting decoded outputs into 3D surfel scenes.

4 Methodology

  • The pipeline encodes reference imagery and depth into latents, generates an extensible canvas, decodes it as RGBD, and lifts it into a 3D scene.Its d-RAE combines a frozen DINOv2 RGB encoder with a separately trained depth ViT, normalizes their concatenated patch tokens, and decodes the fused representation.Source: Figure 1 · p. 5, §4 · p. 6

4.1 Latent Initialization and Conditioning Interpolation

  • Four reference-image embeddings are bilinearly interpolated into a conditioning grid, allowing appearance to vary smoothly without fixing scene structure.Each pooled descriptor conditions a small image region, while the shared latent space preserves coherence.Source: §4.1 · pp. 6–7

4.2 Extensible Generation with Conditional Flow Matching

  • Overlapping latent windows are reconciled by averaging their guided flow trajectories during sampling, so one model fuses neighboring regions without post-hoc stitching.The model learns rectified flow from Gaussian noise to latent representations at a fixed low resolution; inference applies Euler integration to overlapping 16 × 16 windows.Source: §4.2 · pp. 7–8

4.3 Latent Decoding and 3D Lifting

  • The fully convolutional decoder produces arbitrary-size RGBD mosaics, which are lifted into 3D as surface-aligned Gaussian surfels for continuous novel-view rendering.Surfel extents follow local pixel spacing and avoid spanning depth discontinuities; appearance refinement adjusts color, opacity, and bounded in-plane scale while geometry remains fixed.Source: §4.3 · p. 8

5 Experiments

Experiments assess single-image generation and reference adherence, extensible landscape generation, and 3D benthic environment rendering.

5.1 Model optimization

  • CFM is trained single-stage for 200 epochs, while d-RAE uses a 16-epoch three-phase reconstruction, perceptual, and adversarial curriculum; both use AdamW and EMA decay 0.999.The curriculum adds LPIPS at epoch 6 and an adversarial term at epoch 8.Source: §5.1 · p. 9

5.2 Single Image Generation

  • FID ≤20 across reefs and preprocessing variants, BenthicFlow outperforms 4-bit FLUX.2-dev in fidelity and conditional similarity.BenthicFlow also preserves reference-reef visual identity, while FLUX.2-dev introduces structural and optical artifacts.Source: Table 1 · p. 9, §5.2 · p. 10, Figure 2 · p. 11
  • BenthicFlow retains over 95% of real CLAHE data’s balanced accuracy and macro F1; BenthicFlow-DPF retains 95% and 94% of real DPF-Net data, respectively.Both variants substantially outperform FLUX.2-dev in predicting source reef, indicating retention of site-specific visual cues.Source: §5.2 · p. 10, Table 2 · p. 11

Window merging

  • The Seam Gradient Ratio compares average gradient magnitudes in a seam band of width S/2 with the rest of the mosaic; a score near 1 indicates no distinguishable seam.The seam band is centered on the overlap boundary, where visible seams tend to occur.Source: Window merging · pp. 10, 12
  • The seam gradient ratio is optimized between 50% and 75% overlap, with 62.5% optimal; performance declines at 87.5% overlap and with no overlap.The study generates 500 terrains per configuration across seven overlap settings, at 16 times the CFM output resolution.Source: Window merging · p. 12, Figure 3 · p. 13

Latent interpolation

  • Linear interpolation between image seeds yields a smooth semantic transition across panorama windows in the DINOv2 embedding visualization.The analysis reduces averaged patch-token embeddings with PCA and t-SNE; embedding proximity correlates with windows’ positions in the panorama.Source: Latent interpolation · p. 12, Figure 3 · p. 13

Depth Consistency

  • Because cropped-image context makes direct monocular-depth comparison inapplicable, d-RAE reconstruction on unseen test pairs provides an indirect consistency check, showing near-perfect depth alignment.The evaluation processes full-sized test images with DAv2 before cropping to bypass the context deficit.Source: Depth Consistency · pp. 12–13, Table 3 · p. 13

3D Rendering

BenthicFlow generates coherent, diverse reef scenes from references across geographic reefs, while DPF-Net improves underwater appearance; the lifting ablation favors surface-aligned surfels with anchored refinement.

3D Rendering

  • At 32 times the training resolution, both variants produce coherent, diverse reef mosaics blending references from different geographic sites.The left scene transitions from sand to rocky structures and includes plating corals and algae; BenthicFlow-DPF broadens color range and improves texture quality by mitigating underwater attenuation.Source: 3D Rendering · pp. 13–14, Figure 4 · p. 14
  • The paired lifting ablation evaluates front-view fidelity on 300 held-out RGBD samples, but does not measure off-axis multi-view coherence.Configurations share identical inputs; the study compares surface-aligned surfels with isotropic 3D Gaussians and ablates appearance refinement.Source: 3D Rendering · p. 14
  • Surface-aligned initialization outperforms isotropic initialization before refinement across all metrics, while perceptual color polishing adds no reprojection benefit and slightly degrades performance.The authors therefore use surfels with anchored refinement and no perceptual polish.Source: 3D Rendering · p. 14, Table 4 · p. 15
  • Anchored refinement reduces LPIPS from 0.138 to 0.046 and raises SSIM from 0.884 to 0.925, improving LPIPS on 94% of samples.The paired evaluation uses 300 generated samples. Isotropic Gaussians reach 0.39 dB higher PSNR after convergence, but the authors adopt surface-aligned surfels with anchored refinement and no perceptual polish for multi-view coherence.Source: 3D Rendering · p. 14, Table 4 · p. 15

6 Conclusions and Future Work

BenthicFlow provides a unified framework for generating extensible 2.5D underwater environments, integrating flow-matching sampling with surface-aligned Gaussian surfels. Future work targets more informative conditioning, direct 3D generation, and higher-resolution synthesis.

6 Conclusions and Future Work

  • The framework combines MultiDiffusion-inspired sampling within a flow-matching trajectory and surface-aligned Gaussian surfels to generate scalable underwater environments for perception and robotics.The surfels improve surface coverage while preserving estimated geometry when lifting single-view RGBD mosaics.Source: §6 · p. 15
  • Future work could improve appearance control with alternative DINOv2 representations, address missing geometry through direct 3D generation, and explore pseudo-3D supervision from VGGT- Ω.The current conditioning uses average-pooled DINOv2 patch tokens, and the pipeline remains fundamentally 2.5D.Source: §6 · p. 15
  • High-resolution synthesis is limited by full-latent-canvas decoding costs; consistent windowed decoding and higher-resolution full-image training may improve global structure and preserve finer details after 3D lifting.These are proposed future directions rather than demonstrated results.Source: §6 · p. 15

S1 Additional Single Image Generation Results

Extended single-image generation results remain consistent across campaigns, while Batemans campaigns perform relatively worse; the authors hypothesize that training-data imbalance contributes to this difference.

S1 Additional Single Image Generation Results

  • Results remain consistent across campaigns, while Batemans campaigns show relatively inferior performance.The extended tables report results for every available campaign.Source: Table S1 · p. 20, S1 Additional Single Image Generation… · p. 20, Table S2 · p. 21
  • Per-campaign tables report generative fidelity and semantic alignment, with depth consistency and reconstruction metrics included for reconstruction evaluations.FID and KID are lower-is-better, DINOv2 cosine similarity is higher-is-better; depth RMSE is lower-is-better and Pearson correlation r is higher-is-better. Reconstruction is evaluated only for the two flow-based pipelines.Source: Table S1 · p. 20, Table S2 · p. 21
  • The authors hypothesize that fewer training images for Batemans campaigns weaken the learning signal relative to Scott Reef and Hawaii.Scott Reef and Hawaii have more images per campaign in Squidle+ [12].Source: S1 Additional Single Image Generation… · pp. 20–21

S2 Extended Analysis of the Extensible Generation

Extended-generation analyses show that optimal overlap reduces visible seams and improves SGR, while sine-weighted merging uniquely achieves an optimal SGR across overlap levels.

S2 Extended Analysis of the Extensible Generation

S3 Benthic Conditioning of FLUX.2-dev

FLUX.2-dev is conditioned on a reference image and a prompt describing key benthic-image properties while encouraging diversity. Its inference setup uses 28 steps, balancing image quality and speed.

S3 Benthic Conditioning of FLUX.2-dev

  • Conditioning uses a 512 × 512 reference image and a prompt describing top-down nadir benthic imagery, substrate types, lighting, attenuation, and style while requesting substrate variation.The prompt asks the model to change the benthic substrate distribution while preserving perspective, lighting, and style.Source: S3 Benthic Conditioning of FLUX.2-dev · p. 23
  • 28 inference steps, following, provide an image-quality and inference-speed trade-off; quantized FLUX.2-dev generates one image per ∼6 seconds, while BenthicFlow generates more than 120 in the same time on the same H100.The FLUX.2-dev rate is for the 4-bit quantized model on an NVIDIA H100 GPU.Source: S3 Benthic Conditioning of FLUX.2-dev · p. 23

S4 Additional Generated Scenes

Additional renderings show BenthicFlow generating large, coherent scenes with smooth transitions between seeds and natural benthic biome variation.

S4 Additional Generated Scenes

  • At 36 times the model-window resolution—6× horizontally and vertically—the generated scenes show smooth seed transitions and natural benthic biome variation.Fig. S3 presents additional scenes generated by BenthicFlow and BenthicFlow-DPF.Source: S4 Additional Generated Scenes · p. 23, Figure S3 · p. 24
  • Fig. S3 shows BenthicFlow and BenthicFlow-DPF scenes with reference seeds sampled from different deployments and campaigns.Source: Figure S3 · p. 24

S5 Dataset Details

The section documents the Squidle+ subset at the deployment level and describes a campaign-wise split that holds out complete deployments for validation and testing.

S5 Dataset Details

  • The selected Squidle+ subset is specified at deployment level, with each deployment’s split and campaign reported to support reproducibility.Source: S5 Dataset Details · p. 25
  • The split uses whole-deployment holdouts per campaign, preventing spatially overlapping imagery across splits; totals are 510,307 train, 76,480 val, and 87,574 test images over 52/7/7 deployments.Validation and test each hold out one complete deployment per campaign, never seen during training.Source: Table S3 · p. 25

Related papers

Loading 2608.23173v1…