PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Visual and Geometric Priors for Metric-scale and Complete Vehicle Gaussian Reconstruction from Limited Views

Jinyu Miao, Jiusi Li, Yifei He, Miao Long, Kun Jiang, Mengmeng Yang, Diange Yang

arXiv:2609.08841v1cs.CV

TL;DR

Limited onboard views often expose only one side of a vehicle, while image-to-3D generation may not provide reliable metric scale. The paper combines a visual foundation model, a learnable Gaussian head, and symmetry-aware Gaussian cloning; experiments report improved completeness and geometric accuracy over existing approaches.

  • Problem

    Vehicle reconstruction from onboard images is challenged by unreliable metric scale in image-to-3D generation and sparse views that leave the opposite side largely unobserved.

    Source: §I · p. 1

  • Method

    A visual foundation model supplies metric geometry for Gaussian initialization, a learnable head estimates Gaussian attributes, and symmetry-aware cloning completes the unobserved side.

    Source: §I · pp. 1–2

  • Results

    Experiments report that the method achieves the best vehicle construction quality among existing baselines in geometric completeness and appearance consistency.

    Source: §I · p. 2

  • Takeaways & Limitations

    The resulting assets combine metric scale, completeness, multi-view consistency, and efficient construction for object-level editing and controllable traffic-scene generation.

    Source: §V · p. 7

Abstract

from arXiv · show

High-fidelity vehicle assets are essential for controllable traffic scene generation, particularly for synthesizing rare and safety-critical long-tail scenarios. However, reconstructing a reusable vehicle representation from in-the-wild onboard images remains challenging for two reasons. First, image-to-3D generation methods generally produce models without reliable metric scale. Second, onboard cameras usually observe only one side of a target vehicle, making conventional multi-view reconstruction incomplete on unobserved regions. To solve these problems, we propose a feed-forward vehicle asset reconstruction method, which leverages two complementary priors to reconstruct 3D Gaussian representations for vehicles using sparse one-sided observations. To achieve metric-scale reconstruction, a visual foundation model is first utilized to serve as a visual prior for Gaussian initialization. The Gaussian attributes are then estimated by a learnable encoder-decoder module. A symmetry-aware cloning strategy is presented to complete the unobserved side directly in Gaussian space, which exploits the bilateral structure of vehicles as a geometric prior. Experiments on the public dataset demonstrate that the proposed method significantly outperforms existing approaches in both vehicle asset completeness and geometric accuracy.

I. INTRODUCTION

The paper targets reusable vehicle assets from limited onboard views, where generated models may lack metric scale and reconstruction cannot recover largely unseen regions. It combines a visual foundation model with Gaussian prediction and symmetry-aware cloning; experiments report stronger vehicle construction quality than existing baselines.

I. INTRODUCTION

  • Onboard videos provide diverse vehicles, but image-to-3D generation often lacks reliable metric scale while reconstruction misses unobserved regions.Vehicles are important foreground objects for object-centric traffic-scene generation.Source: §I · p. 1
  • A visual foundation model supplies metric-aware geometry for Gaussian initialization, while a trainable Gaussian head predicts Gaussian attributes.The proposed framework is feed-forward and targets sparse one-sided observations.Source: §I · p. 1
  • Symmetry-aware cloning applies vehicle bilateral structure directly in Gaussian space to complete the unobserved side efficiently.The approach clones the well-observed side across the symmetry plane rather than augmenting input images through flipping.Source: §I · pp. 1–2
  • The method achieves the best vehicle construction quality among existing baselines in geometric completeness and appearance consistency.The experiments use real data.Source: §I · p. 2

II. RELATED WORKS

Prior work spans dynamic-scene reconstruction, photorealistic data generation, and object-asset construction, but the cited approaches leave challenges in efficiency, controllability, completeness, metric scale, or appearance consistency. The framework defines feed-forward prediction of a vehicle Gaussian representation from images and poses.

A. Dynamic Traffic Scenes Reconstruction

  • NeRF and 3D Gaussian methods support novel-view traffic-scene reconstruction, while dynamic-scene methods add temporal attributes or separate foreground and background.NeRF approaches typically require auxiliary motion cues and are computationally limited; vanilla 3D Gaussian Splatting assumes a static scene.Source: A. Dynamic Traffic Scenes Reconstruction · p. 2

B. Photorealistic Data Generation

  • BEV-guided video diffusion and object-level editing address driving-data generation, but controllability, appearance consistency, or foreground-background consistency remain challenges.Geometrically complete and appearance-consistent assets are described as essential for realistic augmented driving data.Source: B. Photorealistic Data Generation · p. 2

C. Object Asset Construction

  • Traditional reconstruction yields sparse point clouds, optimization-based NeRF and Gaussian methods take tens of minutes per object, and feed-forward or diffusion methods retain other limitations.Feed-forward performance degrades with sparse onboard views; diffusion-based methods may lack metric-scale accuracy and consistency with real observations.Source: C. Object Asset Construction · p. 2

A. Problem Definition and Framework Overview

  • The framework predicts a target vehicle's 3D Gaussian representation feed-forward from RGB images and corresponding poses.The representation uses Gaussian attributes including covariance, opacity, and spherical-harmonic appearance coefficients, and supports alpha rendering.Source: A. Problem Definition and Framework… · p. 2

B. Metric-Scale Geometry Reconstruction using Visual Prior

A frozen MapAnything visual prior estimates metric-scale geometry from multi-view images, which is aggregated into a colored point cloud to initialize vehicle Gaussians.

B. Metric-Scale Geometry Reconstruction using Visual Prior

  • MapAnything supplies metric-scale depth from a pretrained visual prior, avoiding reliance on sparse-feature MVS or a task-specific network to recover absolute scale.The paper identifies sparse visual correspondences and large-scale training-data requirements as challenges for those alternatives.Source: B. Metric-Scale Geometry… · p. 3
  • From multi-view images and optional poses, MapAnything predicts per-image ray and depth maps plus a globally consistent scale for metric point reconstruction.The reconstructed points are transformed from camera coordinates into the vehicle frame.Source: B. Metric-Scale Geometry… · p. 3
  • A vehicle mask filters background pixels before valid points are aggregated into a metric colored cloud that anchors Gaussian centers and colors.The cloud supports insertion into a metric scene without post-hoc scale recovery.Source: B. Metric-Scale Geometry… · p. 3

C. Gaussian Attribute Estimation

A trainable Gaussian head estimates renderable Gaussian attributes while retaining the visual prior’s metric geometry and allowing local corrections.

C. Gaussian Attribute Estimation

  • A ResNet-50 encoder and UNet-style decoder use concatenated RGB and depth inputs to predict per-pixel Gaussian opacity, position offset, rotation, scale, and appearance attributes.The appearance vector contains spherical-harmonic coefficients up to degree 2.Source: C. Gaussian Attribute Estimation · p. 3
  • Gaussian scales are constrained to positive, bounded values by applying Softplus and clipping, while normalized quaternions provide rotation matrices.The clipping operation truncates values outside the specified bounds.Source: C. Gaussian Attribute Estimation · p. 3
  • Position offsets refine visual-prior metric points rather than replacing them, preserving global metric geometry while correcting local errors around wheels, lamps, and mirrors.The center calculation adds the predicted offset after transforming the recovered point into the vehicle frame.Source: C. Gaussian Attribute Estimation · p. 3
  • Observed RGB initializes Gaussian color, allowing basic vehicle color reproduction before higher-order view-dependent effects are learned.The paper states this initialization facilitates network training.Source: C. Gaussian Attribute Estimation · p. 4

D. Symmetry-Aware Gaussian Cloning using Geometry Prior

The method completes one-sided vehicle reconstructions by exploiting approximate bilateral symmetry directly in Gaussian space instead of augmenting images with flips.

D. Symmetry-Aware Gaussian Cloning using Geometry Prior

  • Approximate bilateral symmetry addresses sparse or unreliable Gaussian geometry on the unobserved side of vehicles seen from only one side.The method treats vehicle symmetry as an explicit geometric prior.Source: D. Symmetry-Aware Gaussian Cloning… · p. 4
  • The denser side of the xz-plane is selected as reference, and its Gaussians are flipped by transforming their centers, covariances, and rotations.The flipped Gaussians inherit scale and opacity; spherical-harmonic coefficients for odd azimuthal orders change sign.Source: D. Symmetry-Aware Gaussian Cloning… · p. 4
  • Merging reference and flipped Gaussians completes the asset without doubling encoded views, while preserving nearly the same rendering quality.The paper contrasts cloning with image flipping, which increases computational burden during training and inference.Source: D. Symmetry-Aware Gaussian Cloning… · p. 4

E. Training scheme

Training uses one-sided input views and opposite-side novel-view supervision, optimizing rendered-image similarity while freezing the visual foundation model.

E. Training scheme

  • Each iteration uses N images from one vehicle side as input and randomly samples two opposite-side target views, excluding cameras within 0.5 meters of mirrored input poses.The exclusion avoids trivial correspondence with horizontally mirrored inputs.Source: E. Training scheme · p. 4
  • The predicted full Gaussian representation is rendered from each target camera, and training combines pixel-wise MSE with LPIPS weighted by 0.05.The loss compares each rendered target view against its target image.Source: E. Training scheme · p. 4
  • The visual foundation model remains frozen while only the Gaussian attribute network is optimized, preserving pretrained visual priors and reducing trainable parameters, memory, and training time.Source: E. Training scheme · p. 4

A. Dataset and Preprocessing

Experiments use 3DRealCar vehicles and simulate sparse onboard observations by reconstructing each test vehicle from four images from one side. Evaluation compares opposite-side novel views across three rendering metrics and representative reconstruction or generation baselines.

A. Dataset and Preprocessing

  • 45 3DRealCar vehicles are split into 25 training and 20 test vehicles, with rendered images at 512 × 384 resolution.COLMAP camera poses are aligned to scanned point clouds to scale them to metric units.Source: A. Dataset and Preprocessing · p. 4

B. Experimental Setup

  • Each test vehicle is reconstructed from four images from one side and evaluated at eight opposite-side viewpoints using PSNR, SSIM, and LPIPS.Source: B. Experimental Setup · p. 4
  • The baselines are per-vehicle optimization with 3D GS, aggregated per-image Gaussians from DA3, and multi-image-conditioned generation with TRELLIS.TRELLIS outputs are aligned to a metric COLMAP point cloud through a scale factor before evaluation.Source: B. Experimental Setup · pp. 4–5
  • The proposed model is trained for 300 iterations on an NVIDIA Tesla V100 GPU with batch size 4 vehicles, using AdamW at learning rate 10−4; its visual foundation model uses MapAnything pretrained weights and remains frozen.Source: B. Experimental Setup · p. 5

C. Comparison with State-of-the-Art Methods

On opposite-side novel views, the proposed method outperforms the compared methods across PSNR, SSIM, and LPIPS. Its rendering quality is also comparable for left- and right-side inputs.

C. Comparison with State-of-the-Art Methods

D. Ablation Studies

The ablations examine position refinement and Gaussian cloning. Learnable offsets improve rendering metrics, while cloning offers a faster alternative to image flipping with comparable rendering quality.

1) Gaussian Position Offset:

  • Learnable Gaussian position offsets consistently improve all evaluation metrics over using visual-foundation-model geometry alone.The initial geometry is satisfactory but contains mismatches; offsets help represent finer vehicle geometry.Source: 1) Gaussian Position Offset · p. 6

2) Gaussian Cloning v.s. Image Flipping:

  • 1.547 s for Gaussian cloning is approximately half the 3.032 s required by image flipping, with comparable rendering quality.Image flipping achieves slightly higher rendering quality but doubles the number of input images.Source: 2) Gaussian Cloning v.s. Image Flipping · p. 6

3) Activation Functions for Scale Parameterization:

Softplus is the best tested activation for Gaussian scale parameterization. Real-world editing visualizations and the paper’s conclusion present the reconstructed assets as complete, metric-scale, and suitable for object-level editing and controllable traffic-scene generation.

3) Activation Functions for Scale Parameterization:

  • Softplus achieves the best performance among Softplus, Sigmoid, and ReLU for Gaussian scale parameterization.Its smooth, strictly positive output enforces valid Gaussian scales while facilitating stable optimization.Source: 3) Activation Functions for Scale… · p. 7

E. Real-world Object-level Editing

  • Real-world editing visualizations show geometrically complete, metric-scale, multi-view-consistent assets reconstructed from limited onboard observations.The assets are inserted into onboard videos using G2Editor and rendered from specified novel viewpoints.Source: E. Real-world Object-level Editing · p. 7

V. CONCLUSION

  • The conclusion describes a dual-prior framework: visual geometry supplies metric-scale Gaussian anchors, bilateral symmetry supports cloning to complete unseen regions, and a trainable head estimates attributes and local corrections.Experiments on 3DRealCar show improvements over existing baselines; the resulting assets are presented as suitable for object-level editing and controllable traffic-scene generation.Source: §V · p. 7

Related papers

Loading 2609.08841v1…