PaperlayerView exact revision on arXiv

Source-linked AI summary

M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis

Yang Zhou, Jiuhong Xiao, Shizhao Ye, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno

arXiv:2609.30056v1cs.ROcs.CV

TL;DR

Robotic novel view synthesis needs visual appearance and metric structure, but generative methods often rely on images alone. M3GD projects frozen image and point-cloud foundation features onto a shared grid and uses LiDAR packets to condition latent generation. On GrandTour it improves RGB and depth synthesis over an image-only backbone, and ground-robot deployment exposes a quality–cost trade-off controlled by Euler steps.

  • Problem

    Generative NVS commonly conditions on posed images alone, although robotic platforms often have calibrated LiDAR and need both visual appearance and metric structure.

    Source: §I · p. 1

  • Method

    M3GD combines explicit geometry statistics and learned point-cloud descriptors in projected LiDAR packets that condition an image-latent generator without separate cross-modal translator pretraining.

    Source: §I · p. 2

  • Results

    On GrandTour, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone.

    Source: §VIII · p. 17

  • Takeaways & Limitations

    A GrandTour-trained M3GD model runs on a ground robot with a different sensor suite without retraining, while Euler-step count trades latency for perceptual quality.

    Source: §VIII · p. 17

    Reported runtimes use an RTX A6000 and do not guarantee real-time performance on smaller onboard computers; single-step sampling substantially degrades LPIPS on the deployment route.

    Source: §VII · p. 16

Abstract

from arXiv · show

Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned LiDAR content and that target-view LiDAR acts as a geometric query linking the requested view to source observations. Deployment on a ground robot demonstrates practical real-world operation, with a configurable quality--cost trade-off controlled by the number of Euler integration steps.

I. Introduction

Robotic NVS must combine visual appearance with metric structure, but generative methods commonly condition on posed images alone. M3GD projects frozen image and point-cloud features onto a shared grid, using LiDAR packets to condition generation without separately pretraining a cross-modal translator.

I. Introduction

  • Robotic NVS can use synchronized LiDAR near the queried pose even when target RGB is unavailable, treating the sweep as a geometric query.The query provides sparse metric evidence without revealing the held-out appearance.Source: §I · p. 1

A. Generative Novel View Synthesis in Geometry-Aware Latents

  • Earlier generative methods condition on RGB and poses, while LiDAR-assisted approaches commonly fit or update scene-specific representations before querying novel views.GLD generates in a frozen geometry-aware image latent but remains image-input-only; M3GD extends its conditioning to measured 3D point clouds.Source: §I · p. 1, A. Generative Novel View Synthesis in… · p. 2

I. Introduction

  • Projected frozen point-cloud and image features share substantial spatial structure, motivating their composition as a cross-modal representation.Packets combine explicit metric geometry statistics with pooled learned point-cloud descriptors on the image-latent grid.Source: §I · p. 2
  • M3GD conditions generative NVS on explicit geometry statistics and learned point-cloud descriptors without separately pretraining a cross-modal translator.The method combines the two signals in a projected feature representation and injects it into an image-latent generator.Source: §I · p. 2
  • M3GD improves target-view RGB and depth synthesis on GrandTour over an image-only version of the same backbone.The paper also reports deployment on a ground robot with a different camera–LiDAR suite and no retraining on the target domain.Source: §I · p. 2

B. Generalizable Feed-Forward View Synthesis

Image-only feed-forward methods infer scene geometry from images, while many depth- and LiDAR-assisted approaches optimize scene-specific representations. M3GD instead composes independently trained 2D and 3D features after projection, using measured LiDAR as reusable conditioning for generative NVS.

B. Generalizable Feed-Forward View Synthesis

  • Feed-forward view-synthesis models derive geometry from images and do not consume pretrained point-cloud representations at inference time.M3GD instead iteratively denoises a target latent conditioned on images and projected LiDAR features.Source: B. Generalizable Feed-Forward View… · p. 2

C. Depth and LiDAR Assisted View Synthesis

  • Many depth- and LiDAR-assisted rendering methods construct, optimize, or update an object- or environment-specific model before querying novel views.M3GD keeps LiDAR as structured conditioning and avoids scene-specific fitting at deployment.Source: C. Depth and LiDAR Assisted View… · p. 3

D. Geometric-Aware Foundation Models and Data Alignment

  • The paper compares projected Utonia features with DA3 and DINOv2 image features on spatially overlapping grids and reports substantial shared spatial structure.It presents measuring this similarity and using it for generative NVS as contributions not previously reported to the authors’ knowledge.Source: D. Geometric-Aware Foundation Models… · p. 3
  • M3GD asks whether independently trained 2D and 3D foundation representations already share enough structure after projection to be composed without explicit image–point alignment training.This differs from cross-modal distillation approaches that supervise a 3D encoder to reproduce 2D features.Source: D. Geometric-Aware Foundation Models… · p. 3

A. Problem Setup

The task is to synthesize target-view RGB and dense depth from posed source RGB–LiDAR observations and target camera poses, with a nearby target LiDAR sweep available but no target image input. The model generates image latents using conditional flow matching.

A. Problem Setup

  • The target-view RGB images and dense depth maps are predicted from posed source observations and target cameras, with target LiDAR potentially available but target RGB unavailable.The target sweep serves as sparse metric evidence without supplying the held-out RGB image or depth map.Source: A. Problem Setup · p. 4

B. Diffusion and Flow Matching

  • Flow matching generates a multi-view latent by defining a path between the clean latent representation and pure noise.The generated latent is later decoded into RGB and depth.Source: B. Diffusion and Flow Matching · p. 4
  • At inference, the model integrates its learned velocity field backward from noise to the clean latent, typically with an Euler solver.The conditioning variable contains source images, camera geometry, and projected LiDAR packets.Source: B. Diffusion and Flow Matching · p. 4

C. 2D and 3D Foundation Feature Backbones

M3GD pairs geometry-aware image latents from DA3 with learned per-point descriptors from Utonia. Projection places LiDAR evidence on the image-latent grid, where a residual adapter adds it to the inherited multi-view generator’s conditioning.

C. 2D and 3D Foundation Feature Backbones

  • DA3 supplies dense image-token features aligned with local image regions, whose depth, ray, and multi-view objectives make them geometry-aware as well as appearance-aware.M3GD uses the encoded representation before DA3’s final depth prediction.Source: C. 2D and 3D Foundation Feature… · p. 4
  • Utonia supplies frozen geometry-first per-point descriptors from a self-supervised encoder trained across heterogeneous point-cloud domains without image supervision.Each learned feature remains attached to its point’s metric location.Source: C. 2D and 3D Foundation Feature… · p. 4

A. Overview

  • Camera projection pools Utonia descriptors into the DA3 image-latent grid, pairing each cell’s image latent with a geometry descriptor and zeroing cells without LiDAR support.A lightweight residual adapter injects view-aligned LiDAR packets while keeping the GLD latent space, decoders, camera conditioning, and training objective unchanged.Source: A. Overview · p. 5

B. Image-Latent Generative Backbone

  • GLD’s multi-view generator receives conditioning features alongside noisy latents, with camera geometry supplied through its existing ray embeddings and multi-view attention.M3GD extends the per-view conditioning feature with spatially aligned LiDAR evidence, including for target views.Source: B. Image-Latent Generative Backbone · p. 6

C. Projected RGB–LiDAR Representation

M3GD projects each calibrated LiDAR sweep onto the image-latent grid, building view-aligned packets from explicit geometry statistics and pooled point descriptors. Target-view LiDAR serves as an appearance-free geometric query, but projection does not reason about visibility.

C. Projected RGB–LiDAR Representation

  • The nearest-timestamp LiDAR sweep is transformed into the camera frame and projected into latent cells using calibrated camera geometry.Each cell's support includes projected points with positive camera-frame depth.Source: C. Projected RGB–LiDAR Representation · p. 6
  • Occupied-cell geometry statistics preserve mean range, support validity, depth variation, and the sweep-to-view time offset; empty cells are zeroed.Projection includes every in-frustum positive-depth point, even points from mutually occluding surfaces.Source: C. Projected RGB–LiDAR Representation · p. 6
  • Each packet combines per-cell geometry statistics with averaged frozen Utonia descriptors on the same image-latent grid.The geometry path summarizes metric measurements and sensor support; the learned path pools descriptors from supporting points.Source: C. Projected RGB–LiDAR Representation · p. 6
  • A target packet uses the same construction as a source packet and supplies appearance-free, coarse-grid geometric conditioning without providing target RGB or the depth output.The platform LiDAR stream must cover the queried timestamp; the packet is empty where no point projects.Source: C. Projected RGB–LiDAR Representation · p. 6

D. Residual LiDAR Conditioning

M3GD replaces GLD's conditioning feature with a zero-initialized residual update that mixes image conditioning, the LiDAR packet, and a source/target role signal. This preserves the backbone's feature shape while leaving cross-view reasoning to its existing multi-view attention.

D. Residual LiDAR Conditioning

  • The residual adapter mixes GLD's conditioning feature with the per-view LiDAR packet and role signal while preserving the conditioning feature's shape.For target views, the image-conditioning slot is zero, making the packet the only observation-dependent conditioning aside from camera geometry.Source: D. Residual LiDAR Conditioning · pp. 6–7
  • The adapter is evaluated once per synthesis query and reused across integration steps because its update depends neither on the noisy latent nor on flow time.The update replaces the conditioning feature while leaving the other two model components unchanged.Source: D. Residual LiDAR Conditioning · p. 7
  • At initialization, M3GD is exactly the pretrained image-only GLD, and optimization opens the LiDAR pathway as it reduces synthesis loss.A shared adapter can behave differently for source views with image latents and target views without them.Source: D. Residual LiDAR Conditioning · p. 7
  • Because the adapter is local and lightweight, cross-view use of packet structure must occur inside GLD's existing multi-view attention.The Zero control retains the adapter while removing LiDAR content.Source: D. Residual LiDAR Conditioning · p. 7

E. Flow-Matching Training and Variable-Step Inference

M3GD retains GLD's flow-matching objective and varies inference cost by changing the number of Euler steps. The shared experimental setup fine-tunes a small residual adapter with frozen backbones and independently dropped camera and LiDAR conditioning.

E. Flow-Matching Training and Variable-Step Inference

  • The target-only flow-matching loss performs slightly better than the all-view variant.The joint latent includes source views whose clean latents are already supplied as conditioning.Source: E. Flow-Matching Training and… · p. 7
  • Camera conditioning and LiDAR packets are independently nulled with probability 0.1 during training, allowing inference without a LiDAR sweep.The LiDAR null is the all-zero packet, matching the representation of a view with no projected points.Source: E. Flow-Matching Training and… · p. 7
  • At inference, all view latents are jointly integrated from noise with a K-step Euler solver, then target latents are propagated through frozen DA3 blocks and decoded to RGB and depth.Reducing K lowers network evaluations; increasing it more finely resolves the same learned flow trajectory.Source: E. Flow-Matching Training and… · p. 7

A. Implementation Details

  • The implementation fine-tunes released GLD weights for 50k steps and uses frozen DA3 and Utonia backbones with a zero-initialized adapter adding about 1.1M parameters.The adapter is roughly 0.2% of the transformer; standard inference uses a 50-step schedule, with deployment varying K without retraining.Source: A. Implementation Details · pp. 7–8

B. Dataset and Evaluation Metrics

Evaluation uses GrandTour's synchronized, calibrated camera–LiDAR recordings and a held-out set of eight scenarios, with target-view appearance, depth consistency, and 3D consistency metrics. The study compares sparse-view generative and feed-forward methods and also measures deployment on a Clearpath Warthog.

B. Dataset and Evaluation Metrics

  • GrandTour evaluation uses 40 training recordings and 8 held-out recordings, with 2,400 clips spanning eight scenarios and co-located target LiDAR queries.Training uses three HDR cameras; evaluation uses the front camera, and target RGB is never given to the model.Source: B. Dataset and Evaluation Metrics · p. 8
  • Target views are assessed on RGB appearance, estimated depth self-consistency, and cross-view 3D consistency; depth metrics do not measure absolute metric depth.The depth evaluation derives estimates uniformly from rendered RGB because GrandTour lacks dense ground-truth depth.Source: B. Dataset and Evaluation Metrics · p. 8

C. Baselines

  • Baselines include diffusion-based Matrix3D, MVGenMaster, and GLD, plus feed-forward MVSplat, DepthSplat, and PixelSplat.GLD is evaluated both off-the-shelf and fine-tuned on GrandTour at DA3 level 0.Source: C. Baselines · p. 8

D. Real-Robot Platform

  • Real-robot runtime is measured on a Clearpath Warthog carrying a MultiSense RGB-D sensor and 360-degree Ouster LiDAR, with an onboard RTX A6000 GPU.The robot logs camera and LiDAR streams; target queries use temporally nearest LiDAR and past aligned observations without target RGB.Source: D. Real-Robot Platform · p. 8, Figure 3 · p. 8

VI. Results

The results first test whether independently trained image and point-cloud features share spatial structure after camera projection. On GrandTour, DA3-Base level 0 shows the strongest overall compatibility with projected Utonia features among DA3 levels.

A. Projected 2D–3D Representation Compatibility

  • The comparison measures spatial relationships across shared cells, not coordinate-wise equivalence between image features and learned point-cloud descriptors.The explicit packet component contains handcrafted depth statistics, so the feature-space comparison focuses on the learned component.Source: A. Projected 2D–3D Representation… · p. 9

1) GrandTour Protocol and Diagnostics:

  • The study compares DA3, DINOv2, VGGT, and Utonia features on LiDAR-supported cells, using multiple structural diagnostics and PCA visualizations.The feature-structure study covers 599,774 processed images from 48 recordings and three HDR camera streams on a shared 16 × 24 grid; Fig. 4 shows Utonia and DA3 recovering similar large-scale partitions on supported cells.Source: 1) GrandTour Protocol and Diagnostics · p. 9, Figure 4 · p. 9, Table I · p. 10
  • DA3-Base level 0 achieves the highest CKA and SVCCA-20, and ranks second on PCA channel correlation, Jaccard@10, and affinity correlation among DA3 levels compared with projected Utonia.Higher DA3 levels are competitive on some low-dimensional or neighborhood diagnostics, but have weaker raw-feature alignment, especially in CKA.Source: 2) Findings · p. 10

2) Findings:

On GrandTour, M3GD improves RGB and depth synthesis over image-only and feed-forward baselines, with particularly clear gains on depth metrics. Its depth readout comparison and sampling-cost results qualify how these comparisons are measured and deployed.

2) Findings:

  • DA3-Base level 0 is more compatible with Utonia than DINOv2 or VGGT on the tested feature comparisons, while cross-modal similarity does not imply image–image feature equivalence.The cross-modal pair remains on a broad scale with the image-only reference and slightly exceeds it on Jaccard@10; M3GD learns its adapter end-to-end without a separate translator or auxiliary alignment objective.Source: 2) Findings · p. 10

B. Baseline Comparison

  • M3GD improves over every baseline on 2D appearance, ranking first on seven of eight metrics, while its margin over fine-tuned image-only GLD is modest.M3GD ties GLD∗† for lowest reprojection error at 0.059, slightly beats PixelSplat’s 0.061, and ranks second on MEt3R behind Matrix3D.Source: B. Baseline Comparison · p. 10
  • M3GD lowers AbsRel by 16% (0.145 vs. 0.173) and RMSE by 11% (0.314 vs. 0.354), and raises the δ<1.25 inlier ratio from 0.777 to 0.810.The depth metrics are computed uniformly from rendered RGB through the RGB readout; all three splat baselines exceed 0.31 AbsRel and 0.76 RMSE.Source: B. Baseline Comparison · p. 10, Table II · p. 11
  • The RGB readout gives uniformly lower M3GD∗† depth error than the native readout, but the method ranking is unchanged across readouts.The RGB readout decodes and re-encodes the latent; the paper attributes its lower error to smoothing rather than a more accurate readout.Source: B. Baseline Comparison · p. 11, Table III · p. 11
  • M3GD is 1.7× faster and uses 33% less memory than GLD at the same resolution and sampling budget.Against Matrix3D, M3GD is 7.6× faster; it matches MVGenMaster latency within run-to-run variation and uses roughly 40% less memory than either.Source: B. Baseline Comparison · p. 11

C. Qualitative Analysis

Qualitative results show that projected target-view LiDAR helps anchor generated views when source images leave target-facing geometry ambiguous. Mechanism ablations find benefits across training modes, while also showing that LiDAR does not replace backbone adaptation.

C. Qualitative Analysis

  • Splatting baselines often show warped or ghosted renderings and distorted depth, while diffusion baselines can show viewpoint hallucinations or target-view-inconsistent geometry.M3GD better anchors target-view layout when source images are ambiguous or do not reveal the target-facing surface, as reflected in RGB and depth visualizations.Source: C. Qualitative Analysis · pp. 11, 13, Figure 5 · p. 12

D. Mechanism Ablations

  • LiDAR conditioning improves every metric in every training mode, with per-mode PSNR gains from +0.63 to +1.72 dB.Under the frozen-backbone setting, the gain is +1.72 dB PSNR, +0.054 SSIM, and −0.074 LPIPS; LiDAR lifts PSNR from 17.39 to 19.11, still below trained variants.Source: Table V · p. 13, 1) LiDAR Conditioning and Training Mode · p. 13
  • The ablations test whether LiDAR helps across training modes, whether real per-sample LiDAR content is needed, and which packet components carry the signal.Fine-tuning from pretrained GLD slightly exceeds training from scratch with LiDAR (22.22 vs. 22.16 PSNR); the frozen-backbone gain is attributable to LiDAR rather than added model capacity.Source: D. Mechanism Ablations · p. 13, 1) LiDAR Conditioning and Training Mode · p. 13

1) LiDAR Conditioning and Training Mode:

M3GD’s LiDAR packet improves hard-view synthesis through per-sample geometry, while learned point descriptors refine appearance. The paper also reports coherent predictions across varied scenes and zero-shot operation on a physical robot.

2) LiDAR Input Mode:

  • On the hard subset, full-packet PSNR is 21.00 vs. 20.31 for no LiDAR; Zero and Learnable param reach 20.37, while geom-only reaches 20.90.The controls show that the gain depends on per-sample LiDAR content, and geometry statistics alone recover most of it.Source: Table VI · p. 13, 2) LiDAR Input Mode · p. 13

1) LiDAR Conditioning and Training Mode:

2) LiDAR Input Mode:

  • Across indoor, semi-open, and outdoor samples, dominant structures remain coherent; artifacts cluster around fine structures, reflective regions, and dynamic objects.The examples use two source views and target-view LiDAR, without target RGB.Source: 2) LiDAR Input Mode · pp. 13–14, Figure 6 · p. 14
  • With large viewpoint changes, geom-only recovers dominant target structure over image-only GLD∗†, while adding Utonia features improves texture and local appearance.The source views may show target-facing regions weakly or not at all.Source: 2) LiDAR Input Mode · pp. 14–15

E. Deployment Recipe Ablations

  • The deployment recipe treats Euler-step budget K and joint target count M as quality–cost choices, profiling runtime and memory alongside quality.These choices establish an operating point under the deployment budget.Source: E. Deployment Recipe Ablations · p. 15, Table VII · p. 16

F. Real-World Robot Deployment

  • On Warthog, the GrandTour-trained model predicts target RGB and depth without retraining, despite LiDAR shadows and residual camera–LiDAR misalignment.The target packet uses the sweep nearest the query timestamp; target RGB is not used as input.Source: F. Real-World Robot Deployment · p. 15, Figure 8 · p. 15

G. Onboard Runtime Tradeoff

M3GD improves GrandTour target-view RGB and depth synthesis over an image-only backbone, with the largest gains on depth metrics. On-robot results show adjustable latency and quality, alongside limits in timing, scene dynamics, evaluation scope, and hardware transfer.

G. Onboard Runtime Tradeoff

  • At K=5, PSNR is 15.86 dB, SSIM 0.281, LPIPS 0.480, and latency 3.63 s; at K=1, two-target latency is 3.13 s but LPIPS degrades to 0.889.At K=10, latency per target falls from 2.13 s at M=2 to 1.21 s at M=4; more steps increase latency without improving appearance metrics on this route.Source: G. Onboard Runtime Tradeoff · p. 16

VII. Limitations and Future Works

  • The evaluation selects the nearest LiDAR sweep but does not isolate camera–LiDAR delay; timing uncertainty remains a future direction.The authors propose a controlled time-offset ablation and explicit modeling of timing uncertainty.Source: §VII · p. 16
  • Per-cell averaging mixes depth layers near object boundaries and thin structures, motivating front-surface pooling or depth-sorted visibility weighting.The residual adapter was kept minimal to make the ablations attributable.Source: §VII · p. 16
  • The benchmark and representation study use GrandTour recordings from one robot platform, ANYmal-D; broader cross-camera, platform, and driving-domain transfer remains to be evaluated.The comparison with feed-forward splatting baselines also cannot perfectly equalize pretraining corpora.Source: §VII · p. 16
  • Fast-moving objects can violate the time-nearest-sweep assumption even at small ∆t, and the model does not explicitly address object motion.Source: §VII · p. 16
  • Single-step sampling substantially degrades LPIPS, and runtimes measured on an RTX A6000 are not guaranteed on smaller onboard computers.Reducing Euler steps lowers latency, trading inference time against perceptual quality.Source: §VII · p. 16

VIII. Conclusion

  • On GrandTour, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone, with the largest gains on depth metrics.The conclusion attributes viability to projected spatial structure shared by frozen image and point-cloud features, while leaving the generator’s latent space and training objective intact.Source: §VIII · p. 17
Loading 2609.30056v1…