PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving

Jiahao Wang, Bo Sun, Yijing Bai, Vincent Casser, Songyou Peng, Zehao Zhu, Meng-Li Shih, Xander Masotto, Shih-Yang Su, Kanaad V Parvate, Tiancheng Ge, Linn Bieske, Dragomir Anguelov, Mingxing Tan, Chiyu Max Jiang

arXiv:2605.22809v2cs.CV

TL;DR

Autonomous-driving systems need diverse long-tail data, but monocular in-the-wild videos lack the multi-view and LiDAR modalities required by target vehicle embodiments. Sensor2Sensor converts such videos into synchronized multi-modal AV logs using synthetic pairing and conditional diffusion, achieving state-of-the-art cross-embodiment generation and realistic conversions of challenging footage.

  • Problem

    In-the-wild monocular videos offer diverse long-tail driving data but lack the multi-view and LiDAR modalities needed for per-embodiment ADS evaluation.

  • Method

    Sensor2Sensor uses 4D Gaussian Splatting to create synthetic dashcam–AV-log pairs and conditional diffusion to convert monocular videos into target-embodiment multi-modal sensor logs.

  • Results

    Sensor2Sensor achieves state-of-the-art cross-embodiment sensor generation; on fixed-camera-to-AV generation, it outperforms all baselines with FID 6.47 and LPIPS 0.316.

  • Takeaways & Limitations

    The method converts challenging internet and dashcam footage into realistic, usable multi-modal sensor logs for AV development.

  • Takeaways & Limitations

    Temporal drift remains a challenge for long-horizon sequences exceeding 30 seconds, potentially reducing long-range temporal coherence and sensor-calibration consistency.

Abstract

from arXiv · show

Robust training and validation of Autonomous Driving Systems (ADS) require massive, diverse datasets. Proprietary data collected by Autonomous Vehicle (AV) fleets, while high-fidelity, are limited in scale, diversity of sensor configurations, as well as geographic and long-tail-behavioral coverage. In contrast, in-the-wild data from sources like dashcams offers immense scale and diversity, capturing critical long-tail scenarios and novel environments. However, this unstructured, in-the-wild video data is incompatible with ADS expecting structured, multi-modal sensor inputs for validation and training. To bridge this data gap, we propose Sensor2Sensor, a novel generative modeling paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds. A core challenge is the lack of paired training data. We address this by converting real AV logs into dashcam-style videos via 4D Gaussian Splatting (4DGS) reconstruction and novel-view rendering. Sensor2Sensor then utilizes a diffusion architecture to perform the generative conversion. We perform comprehensive quantitative evaluations on the fidelity and realism of the generated sensor data. We demonstrate Sensor2Sensor's practical utility by converting challenging in-the-wild internet and dashcam footage into realistic, multi-modal data formats, further unlocking vast external data sources for AV development.

1. Introduction

Sensor2Sensor addresses the embodiment gap between physically grounded in-the-wild videos and the precise multi-modal sensor formats required for ADS validation. It creates paired training data from AV-log reconstructions and uses conditional diffusion to convert monocular dashcam videos into target-embodiment camera and LiDAR logs.

  • Motivation: In-the-wild videos provide physically grounded, long-tail scenarios but remain incompatible with ADS because of a severe embodiment gap.Generative scenario synthesis instead faces non-physical dynamics and low sensor fidelity.
  • Approach: Sensor2Sensor converts in-the-wild monocular videos into high-fidelity, multi-modal AV logs for a target vehicle embodiment.The paradigm combines real-world plausibility with the precise sensor format of the target embodiment.
  • Synthetic Pairing: To overcome missing paired dashcam–AV-log data, the method reconstructs dynamic scenes from existing AV logs and renders realistic dashcam views with sampled camera-parameter augmentations.This produces paired synthetic dashcam and real AV-log training examples.
  • Model: A conditional diffusion model generates eight camera views plus LiDAR from the input dashcam video.The architecture is designed for multi-sensor and multi-modal output.
  • Evaluation: Quantitative fidelity evaluation with manually collected ground truth and qualitative tests on challenging real-world videos show state-of-the-art fidelity and usable sensor-log conversion.The evaluation supports unlocking previously incompatible external data sources for AV development.

2. Related Works

Prior work uses generative world models and diffusion architectures to synthesize photorealistic, physics-based data and consistent multi-sensor outputs. Sensor2Sensor extends this capability to cross-embodiment conversion from monocular video to geometrically accurate multi-sensor AV logs.

  • Generative World Models and High-Fidelity Sensor Synthesis: Generative world models, often diffusion-based, synthesize photorealistic, physics-based data for physical AI.Examples include Wayve’s GAIA-1 and NVIDIA Cosmos.
  • Generative World Models and High-Fidelity Sensor Synthesis: Camera-to-LiDAR models address the spatial and modal mismatch between camera views and 3D point clouds.LiDMs are cited as an example of this generation task.
  • Generative World Models and High-Fidelity Sensor Synthesis: Cross-modality frameworks generate consistent multi-sensor data, while Sensor2Sensor translates a monocular stream into a geometrically accurate, multi-sensor AV log.This cross-embodiment translation motivates a geometrically anchored training corpus.

3. Method

Sensor2Sensor builds paired training data by reconstructing autonomous-driving scenes with 4D Gaussian Splatting and rendering dashcam-style views, then uses a diffusion model to generate synchronized multi-view images and LiDAR from a single camera input. The method further supports temporally consistent video conversion through autoregressive generation with DAgger-based training.

  • 3.1 Synthetic Paired-Data Curation: Approximately 100,000 10-second AV scenes are reconstructed with dynamic and deformable objects, using 360-degree multi-view cameras and LiDAR for geometry initialization and regularization.The reconstructed scenes are rendered from virtual cameras with augmented intrinsic and extrinsic parameters to mimic in-the-wild dashcams.
  • 3.1 Synthetic Paired-Data Curation: The paired-data pipeline reconstructs 4DGS from 8-view cameras and renders synthetic third-party camera views, including popular dashcam models.Virtual-camera rendering uses augmented camera intrinsics and extrinsics, with best quality within a bounded region around the original camera poses.
  • 3.2 Multi-Sensor, Multi-View Generation: The diffusion model simultaneously generates multi-view images and LiDAR point clouds using modality-specific VAEs and U-Net branches with cross-sensor and multi-view attention.Cross-sensor attention enables information exchange between image and LiDAR features, while 3D attention operates across views and spatial locations.
  • 3.2 Multi-Sensor, Multi-View Generation: Camera raymaps condition image generation on camera pose, while a ninth conditional view provides the third-party camera input separately from the N = 8 generated views.The conditional view is encoded and concatenated with its raymap and a binary mask indicating a known, noise-free condition.
  • 3.3 Autoregressive Video Generation: For video conversion, the model autoregressively predicts each timestep’s multi-view images and LiDAR conditioned on the third-party frame and previous self-generated sensors, while DAgger reduces train-test mismatch.DAgger augments training contexts with the model’s own generations to mitigate drifting and accumulated errors during long rollouts.

4. Experiments

Sensor2Sensor is evaluated for multi-view image, video, and LiDAR generation against baselines, with additional tests of in-the-wild generalization and architectural choices. It achieves strong quantitative and qualitative results, including temporal stability, cross-modal LiDAR quality, and preference on challenging external footage.

  • Multi-view image generation: On the “Fixed-Camera-to-AV” image-generation task, Sensor2Sensor outperforms all baselines with FID 6.47 and LPIPS 0.316.Generated images are clear, geometrically plausible, and consistent across camera views, whereas baselines often show blur, distortions, or artifacts.
  • Multi-view video generation: 278.12 FVD is Sensor2Sensor’s best video score, exceeding Ours wo VC (293.73), π3 (2007.35), and VGGT (2373.15).It also achieves PSNR 22.42, SSIM 0.623, and LPIPS 0.186, with smooth, coherent sequences rather than flickering or inconsistent object appearances.
  • LiDAR generation: Sensor2Sensor jointly generates LiDAR and multi-view video with cleaner geometry, fewer noise artifacts, and more accurate intensity rendering than X-Drive and Ours wo VC.Chamfer Distance and human evaluation are reported for LiDAR, with human raters preferring Sensor2Sensor’s generated point clouds.
  • In-the-wild generalization: Sensor2Sensor converts unseen internet, dashcam, and third-party videos into coherent multi-sensor AV logs despite unknown sensor characteristics, environments, weather, and long-tail incidents.Qualitative examples include night-time near-collisions, accidents, crashes, and active incidents while preserving critical scene elements.
  • In-the-wild generalization: 83.46% and 68.08% are Sensor2Sensor’s dashcam top-preference rates for image and LiDAR generation, respectively; internet rates are 84.62% and 58.46%.The evaluation used 26 participants and 40×3 generated image-and-LiDAR samples, judged for realism and alignment with the input.
  • Model Architecture: In the image-only ablation, view concatenation achieves FID 6.20 versus 6.63 for channel concatenation; joint training achieves LPIPS 0.316 versus 0.346 for the CC variant.The full model remains competitive with image-only VC at LPIPS 0.307, supporting joint LiDAR generation without obvious image-quality degradation.

5. Conclusion

Sensor2Sensor bridges the embodiment gap between consumer driving videos and the multi-modal sensor suites required for autonomous-vehicle validation. It uses 4DGS-based data pairing and conditional diffusion to generate synchronized multi-view camera streams and LiDAR point clouds, achieving state-of-the-art cross-embodiment sensor generation.

  • Sensor2Sensor bridges the embodiment gap between consumer driving videos and complex multi-modal sensor suites for AV validation.The paradigm targets the incompatibility between third-party driving videos and validation-oriented sensor configurations.
  • Its pipeline combines 4DGS-based data pairing with a conditional diffusion architecture.These components support the conversion from monocular third-party videos to structured sensor outputs.
  • Sensor2Sensor converts monocular third-party videos into synchronized multi-view camera streams and LiDAR point clouds.The generated outputs form a synchronized, multi-modal sensor suite for AV applications.
  • Sensor2Sensor achieves state-of-the-art performance in cross-embodiment sensor generation.

Supplementary Material · A. Extended Qualitative Results

The supplementary section provides an in-depth visual analysis complementing the main paper’s quantitative results. It evaluates Sensor2Sensor’s efficacy and generalization across image generation, LiDAR generation, and image–LiDAR alignment.

  • A. Extended Qualitative Results: The visual analysis is intended to complement the quantitative results presented in the main paper.The figures provide qualitative evidence alongside the main paper’s quantitative evaluation.
  • A. Extended Qualitative Results: The figures are designed to highlight Sensor2Sensor’s efficacy across different output modalities.The supplementary analysis focuses on how effectively the pipeline produces multiple output types.
  • A. Extended Qualitative Results: The figures are designed to highlight Sensor2Sensor’s generalization capabilities across different output modalities.The qualitative results examine generalization beyond a single output modality.
  • A. Extended Qualitative Results: Additional qualitative results cover image generation in Figures 11 and 12.These figures specifically address the image-generation output modality.
  • A. Extended Qualitative Results: Additional qualitative results cover LiDAR generation in Figure 13.This figure specifically addresses the LiDAR-generation output modality.
  • A. Extended Qualitative Results: Additional qualitative results cover image–LiDAR alignment in Figure 14.This figure specifically addresses alignment between the generated image and LiDAR modalities.

B. Implementation Details · B.1. Training Pipeline

The model is trained through a four-step pipeline that progressively incorporates more complex conditioning information. It begins with single-frame generation and then adds previous-frame camera and LiDAR representations alongside an additional dashcam view.

  • B.1. Training Pipeline: The training pipeline uses four steps to progressively incorporate increasingly complex conditioning information.The supplied passage describes the overall pipeline as four-step, but does not provide the fourth step's details.
  • B.1. Training Pipeline: Step 1 trains single-frame generation conditioned on dashcam images.This establishes the initial generation task before additional temporal and multimodal signals are introduced.
  • B.1. Training Pipeline: Step 1 uses conditional dashcam images as the model input.The passage specifies dashcam images as the conditioning signal for single-frame generation.
  • B.1. Training Pipeline: Step 2 fine-tunes the model with dense conditioning signals from the previous frame.This stage follows the initial single-frame training stage.
  • B.1. Training Pipeline: The previous-frame conditioning includes latent representations of camera and LiDAR data.The passage identifies both modalities as part of the previous frame's latent conditioning.
  • B.1. Training Pipeline: Step 2 also incorporates an additional dashcam view during fine-tuning.This view is listed alongside the previous frame's camera and LiDAR latent representations.

B.2. Model Architecture

Sensor2Sensor uses a conditional diffusion model with a multi-stream UNet for multi-modal sensor data. Separate camera and LiDAR streams learn modality-specific representations while shared attention layers fuse information, with pretrained VAEs and dashcam conditioning supporting compact latent generation.

  • Backbone: The generative model is a conditional diffusion model built around a multi-stream UNet backbone for multi-modal sensor data.The backbone includes temporal attention connections.
  • Backbone: Separate camera and LiDAR streams learn modality-specific representations and fuse information through shared attention layers.The UNet processes 8 surrounding camera views, one dashcam view, and top-mounted LiDAR.
  • Backbone: The UNet uses output channels of (320, 640, 1280, 1280) and a LiDAR stream with 16 input and output channels.The LiDAR channel configuration matches the LiDAR VAE latent space.
  • Variational Autoencoders: Separate pretrained VAEs encode camera views into 8-channel latents and raw LiDAR spin images into a 16-dimensional latent space.The VAEs encode raw sensor data into a compact latent space.
  • Conditioning Mechanisms: Dashcam frames condition diffusion blocks by concatenating features with denoising latents in the view dimension, with random spatial masking applied at probability 0.2 during training.At inference, targeted masks can cover distractors such as dashcam watermarks or the ego vehicle.

B.3. Training Details

Training uses a multi-step finetuning pipeline on 128 TPUs with AdamW, gradient clipping, conditioning dropout, and EMA evaluation. Additional qualitative results show high-fidelity image and LiDAR generation with improved geometry and cross-modal spatial alignment.

  • Optimization: The model is trained on 128 TPUs using AdamW with a 5e-5 learning rate and global gradient-norm clipping at 1.0.Conditioning signals are randomly dropped for regularization.
  • Image Generation: Qualitative image results show superior fidelity, preserving object shape and color, especially for vehicles, compared with the input dashcam image and baselines.The results emphasize vehicle appearance, which is described as challenging for baselines.
  • LiDAR Generation: Qualitative LiDAR results produce more accurate, less noisy point-cloud geometry with better correspondence to accompanying image data.This improved fidelity preserves the scene’s underlying spatial relationships more accurately.
  • Cross-Modal Alignment: Generated LiDAR point clouds accurately reflect synthesized image geometry, demonstrating precise spatial registration and cross-modal consistency.The visualization specifically concerns image-LiDAR alignment.

B.4. Dataset Details … B.8. DAgger Training

The appendix describes the datasets, camera-parameter sampling, multi-sensor adaptation, LiDAR VAE objectives, and DAgger-based training used to support Sensor2Sensor. Together, these procedures define paired data construction, configurable rendering, LiDAR representation learning, and autoregressive video-generation stabilization.

  • B.4. Dataset Details: Training uses 100k 10s clips with 8 cameras and top LiDAR for 4DGS reconstruction, while evaluation uses 1K paired 3s sequences.In-the-wild evaluation includes internet videos, ADAS logs, manually captured dashcam footage, and smartphone footage.
  • B.5. Dashcam Parameter Distribution: Camera extrinsics are sampled by vehicle category from category-specific 6-DoF pose distributions, including sedan heights of 1.1–1.3m and forward translations of 2.0–2.5m.Intrinsics come from calibrated real-world dashcams and receive uniform noise, such as ±5% focal-length variation; outputs receive exposure and gamma correction.
  • B.6. Different Target Camera Configurations: Raymap conditioning encodes camera intrinsics and extrinsics, enabling multi-sensor flexibility by learning the mapping between 3D rays and pixel intensities.Adapting to a new platform requires generating 4DGS-based paired data for its target sensor configuration.
  • B.7. LiDAR VAE Training: The LiDAR VAE jointly encodes depth, intensity, and elongation with convolutional encoder and decoder networks and KL-regularized latent representations.Normalized range, intensity, and elongation use L1 reconstruction, while validity uses cross entropy.
  • B.7. LiDAR VAE Training: The LiDAR VAE total loss is a weighted sum of signal reconstruction, validity, LPIPS perceptual, and KL-divergence terms.LPIPS compares learned feature activations, while KL regularization encourages the latent distribution toward a standard normal prior.
  • B.8. DAgger Training: DAgger adapts imitation learning to autoregressive video generation by treating the generation model as a policy that predicts the next frame from previously generated frames.The expert supplies a correct next frame for policy-generated states, addressing temporal inconsistency.
  • B.8. DAgger Training: DAgger training first fits a base model with ground-truth history, then uses autoregressive rollouts as degraded data and randomly substitutes generated history during improved-model training.This exposes the model to errors accumulated from its own predictions.

C. Limitations and Potential Solutions · D. Synthetic Cameras from 4DGS

Sensor2Sensor combines a high-fidelity per-frame diffusion backbone with autoregressive video synthesis, while 4DGS generates photorealistic synthetic dashcam views for paired training. Temporal drift remains a limitation for long-horizon sequences, motivating stronger consistency priors as future work.

  • C. Limitations and Potential Solutions: The multi-modal diffusion model achieves state-of-the-art per-frame generative quality and serves as a high-fidelity backbone for static scenes.
  • C. Limitations and Potential Solutions: Video synthesis extends the single-frame model autoregressively by conditioning each newly generated frame on the previously generated frame.
  • C. Limitations and Potential Solutions: DAgger finetuning mitigates short-term error accumulation, but temporal drift remains a challenge for long-horizon sequences.
  • C. Limitations and Potential Solutions: Stronger priors for object-level and scene-level consistency over time are identified as promising directions for future exploration.
  • D. Synthetic Cameras from 4DGS: 4DGS synthesizes paired training data by simulating third-party camera views, producing photorealistic dashcam images that mimic real-world optical characteristics and environmental complexity.
  • D. Synthetic Cameras from 4DGS: The diffusion model maps artifact-containing synthetic inputs (Isynth) to pristine real sensor data (Oreal), functioning as denoising while learning robust spatial and semantic mappings.
Loading 2605.22809v2…