PaperlayerView exact revision on arXiv

Source-linked AI summary

L4GM: Large 4D Gaussian Reconstruction Model

Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, Huan Ling

arXiv:2406.10324v1cs.CVcs.LG

TL;DR

L4GM addresses the challenge of reconstructing animated 3D assets from monocular video without costly multiview capture or slow iterative optimization. It extends pretrained LGM with temporal modeling, synthetic multiview-video training, and learned interpolation to produce per-frame 3D Gaussians. The model generalizes well to in-the-wild videos and achieves orders-of-magnitude faster inference than existing 4D reconstruction approaches.

  • Problem

    Single-view dynamic reconstruction is ill-posed, while prior approaches may require many views, iterative optimization, or costly asset-authoring procedures.

  • Method

    L4GM extends pretrained LGM with temporal self-attention, initial-frame multiview completion, per-timestep rendering supervision, and learned 4D interpolation.

  • Results

    L4GM generalizes extremely well from synthetic training to in-the-wild videos and achieves state-of-the-art video-to-4D quality while being 100 to 1,000 times faster than other approaches.

  • Takeaways & Limitations

    L4GM provides fast feed-forward reconstruction of dynamic 3D Gaussian sequences from single-view video, including long-video and higher-frame-rate outputs.

  • Takeaways & Limitations

    The model is trained on non-commercial public datasets and is intended for research-only use; the authors also caution against malicious impersonation.

Abstract

from arXiv · show

We present L4GM, the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second. Key to our success is a novel dataset of multiview videos containing curated, rendered animated objects from Objaverse. This dataset depicts 44K diverse objects with 110K animations rendered in 48 viewpoints, resulting in 12M videos with a total of 300M frames. We keep our L4GM simple for scalability and build directly on top of LGM, a pretrained 3D Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from video frames sampled at a low fps and then upsamples the representation to a higher fps to achieve temporal smoothness. We add temporal self-attention layers to the base LGM to help it learn consistency across time, and utilize a per-timestep multiview rendering loss to train the model. The representation is upsampled to a higher framerate by training an interpolation model which produces intermediate 3D Gaussian representations. We showcase that L4GM that is only trained on synthetic data generalizes extremely well on in-the-wild videos, producing high quality animated 3D assets.

1 Introduction

L4GM addresses the costly and slow creation of animated 3D assets by reconstructing dynamic objects from monocular video in a feed-forward model. It builds on large-scale 3D reconstruction and introduces a 4D dataset and temporal modeling, achieving strong generalization and substantially faster inference.

  • Motivation: Animated 3D assets are important for virtual worlds, but creating them requires time-consuming rigging, skinning, and keyframe authoring.Generating animated assets from monocular videos or text is the stated goal, while advanced 4D editing remains out of scope.
  • Motivation: Prior multiview reconstruction methods require many views across time, while score-distillation methods are fragile and can take hours per prompt.These requirements limit the applicability or speed of earlier approaches.
  • Contribution: L4GM is a 4D Large Reconstruction Model that reconstructs a sequence of 3D Gaussians from monocular video in a feed-forward fashion.It extends pretrained LGM with temporal self-attention and trains on multiview videos using per-timestep rendering losses.
  • Contribution: L4GM uses a new dataset of 12 million multiview videos of rendered animated objects from Objaverse and extends LGM to produce temporally consistent per-frame Gaussian representations.An interpolation model upsamples the representation to a higher frame rate.
  • Results: 100 to 1,000 times faster inference accompanies state-of-the-art quality on video-to-4D reconstruction, while synthetic-only training generalizes extremely well to in-the-wild videos.The reported in-the-wild examples include Sora-generated videos and real-world videos from ActivityNet.

2 Related Work

Related work spans feed-forward 3D and dynamic reconstruction, diffusion-guided optimization, and accelerated monocular-video pipelines. L4GM targets the remaining combination of single-view dynamic reconstruction, explicit 4D representation, generalization, and feed-forward speed.

  • Feed-forward reconstruction: Large reconstruction models use large-scale multiview data to predict object-centric 3D representations directly, avoiding lengthy per-scene optimization.LRM is described as a transformer-based model for NeRF reconstruction from sparse posed images.
  • Video-to-4D reconstruction: Single-view dynamic reconstruction is ill-posed, and existing methods rely on templates, accurate depth, or cannot recover occluded regions.These constraints make monocular video reconstruction particularly challenging.
  • Dynamic novel-view synthesis: Dynamic novel-view methods include synthetic-data transformers, depth-dependent extensions, pretrained models requiring test-time finetuning, and human-template approaches.The cited methods differ in whether they provide explicit 3D representations and whether they require additional optimization.
  • Diffusion-guided generation: Score-distillation approaches optimize 3D or 4D representations using image or video diffusion guidance, with variants adding multiview, flow, or pseudo-label supervision.These methods cover text-, image-, and video-conditional generation settings.
  • Fast reconstruction: Monocular-video acceleration methods may first generate multiview videos and then perform optimization, but reconstruction can still take tens of minutes.L4GM instead directly trains a feed-forward 4D reconstruction model.

3 Background

L4GM builds on LGM, which maps posed multiview images to 3D Gaussian ellipsoids through an asymmetric U-Net. For a single image, ImageDream supplies plausible missing views before LGM performs reconstruction.

  • LGM: LGM directly outputs a 3D reconstruction represented as a set of Gaussian ellipsoids from multiview object images.Each Gaussian uses center, scale, quaternion rotation, opacity, and color-feature parameters.
  • LGM: Camera poses are encoded with Plücker ray embeddings and concatenated with RGB inputs before an asymmetric U-Net produces per-pixel Gaussian parameters.The network yields V 14-channel feature maps, with each pixel interpreted as one Gaussian's parameters.
  • Single-image reconstruction: When only one image is available, ImageDream generates plausible multiview completions that are then passed to LGM for reconstruction.This supplies the multiview input expected by the pretrained model.

4 Our Method

L4GM extends pretrained LGM from static multiview reconstruction to dynamic monocular-video reconstruction by combining initial multiview completion, temporal attention, and per-timestep Gaussian outputs. Learned interpolation and autoregressive processing support smoother high-frame-rate and longer-video reconstruction.

  • Method overview: Given a monocular dynamic-object video, L4GM aims to rapidly reconstruct an accurate 4D representation.The method is motivated by extending a pretrained image-based reconstruction model to temporal data.
  • Method overview: L4GM leverages a pretrained LGM because large-scale static 3D pretraining supplies image-based knowledge before learning temporal dynamics.This follows the image-pretraining-then-video-finetuning strategy used in video generation.
  • Method overview: A single initial set of multiview images is sufficient, with temporal self-attention propagating and adapting that information across subsequent video frames.This avoids requiring multiview video inputs at every timestep.
  • Method overview: L4GM reconstructs a set of 3D Gaussians for each timestep and can be fine-tuned as a 4D interpolation model for higher-FPS, smoother motion.Generated multiview images from the first frame and the monocular video are processed together.
  • Initial multiview generation: ImageDream generates four orthogonal views conditioned on the initial video frame to provide the initial multiview input.The generated views are intended to include three views orthogonal to the original input frame.
  • Initial multiview generation: When generated views do not align with the input frame, LGM first reconstructs an initial Gaussian set and renders it from the desired orthogonal views.This procedure addresses mismatched viewing angles between generated views and the input frame.
  • Model architecture: The model replicates initial multiview images and camera poses across time, then adds temporal self-attention after each cross-view attention layer.The setup assumes a static camera and a moving object.
  • Model architecture: Temporal attention rearranges features so each view is processed as an independent video before restoring the original batch-time-view layout.The operation preserves the spatial feature structure while exposing temporal interactions.

5 Objaverse-4D dataset

The Objaverse-4D dataset is built by rendering animated Objaverse objects into multiview video clips, covering diverse motions and camera views. Its construction uses one-second clips with 48 rendered viewpoints under a predominantly zero-elevation camera assumption.

  • Dataset collection renders all animated objects in Objaverse 1.0 for the 4D reconstruction task.
  • 44,000 of 800,000 Objaverse objects have animations, yielding 110,000 total animations.
  • The animations include rigged characters, locomotion, facial expressions, smoke effects, and deforming fantasy-style motions.
  • Each animation is split into one-second subclips and rendered into 48 views.
  • The rendering setup uses 16 fixed cameras at 0° elevation with uniformly distributed azimuths and 32 additional cameras.

6 Experiments

L4GM is evaluated for video-to-4D reconstruction using quantitative benchmarks, qualitative comparisons, user studies, and ablations. It achieves strong reconstruction quality and fast inference, while the studies identify the effects of pretraining, temporal attention, design choices, video length, and interpolation.

  • Quantitative Evaluation: L4GM outperforms existing video-to-4D approaches on all reported quality metrics on the Consistent4D benchmark.The benchmark uses novel-view LPIPS, CLIP similarity, FVD, and runtime evaluation.
  • Qualitative Evaluation: L4GM produces high-quality, sharp renderings with strong temporal and multiview consistency on in-the-wild videos.The qualitative examples come from ActivityNet and Consistent4D videos.
  • Comparisons to State-of-the-Art Methods: L4GM is substantially faster than optimization-based baselines, which require 10 minutes or 2 hours for 64-frame videos.The comparison includes DG4D, STAG4D, and OpenLRM across generated and real-world videos.
  • Comparisons to State-of-the-Art Methods: L4GM is most favorable in the user study across overall quality, 3D appearance, input-video alignment, motion alignment, and motion realism.The study compares synthesized 4D scenes using 24 examples.
  • Ablation Studies: Removing 3D pretraining prevents convergence, while freezing the pretrained LGM layers converges faster initially but reaches lower PSNR.The ablations support end-to-end fine-tuning of the pretrained model for 4D reconstruction.
  • Ablation Studies: Temporal attention improves temporal modeling, whereas its removal lowers PSNR and produces visible flickering in reconstructed novel-view videos.A deformation-field variant also fails to learn useful motion at scale and produces static outputs.
  • Ablation Studies: Longer autoregressive reconstruction reduces quality with more autoregressive runs, while T = 16 provides balanced performance across video lengths.Shorter inputs start with higher quality but degrade faster because they require more self-reconstructions.
  • Ablation Studies: 4D interpolation successfully increases the output framerate beyond the input video framerate.The comparison is performed on an 8-FPS Consistent4D video.

7 Conclusions

L4GM reconstructs dynamic objects as sequences of 3D Gaussians from monocular video, using pretrained 3D knowledge, synthetic dynamic data, and learned interpolation. It supports long-video reconstruction and fast inference, while advanced human-in-the-loop editing remains needed for professional use.

  • Conclusions: L4GM reconstructs dynamic sequences of 3D Gaussians from single-view video using a pretrained 3D model and the Objaverse-4D dataset.The model learns temporal dynamics from synthetic dense-view 4D data.
  • Conclusions: Learned interpolation enables higher-framerate outputs, while the model achieves orders-of-magnitude faster inference than existing 4D reconstruction or text-to-4D methods.The interpolation model operates directly in 4D space because Gaussian trajectories are not tracked across frames.
  • Conclusions: L4GM generalizes to in-the-wild real and generated videos.The conclusion identifies this generalization alongside faster inference as a central result.
  • Conclusions: Advanced human-in-the-loop 4D editing capabilities are still needed to make the technology useful for professionals.The paper identifies this as an example of the remaining challenges.
  • Conclusions: Autoregressive reconstruction extends L4GM to videos substantially longer than its training length.The process can be repeated more than 10 times without a significant quality drop.

B More Implementation Details

The implementation uses rendered Objaverse-4D data, a pretrained LGM backbone, temporal attention, rendering-based losses, and distributed training. Inference runs on a consumer GPU and combines feed-forward reconstruction with learned interpolation for higher frame rates.

  • Dataset and model: The dataset is rendered with Blender's EEVEE engine using random or fixed camera settings aligned with LGM's pretrained setup.Rendering the dataset requires three days with 200 GPUs.
  • Dataset and model: Motion filtering retains clips whose average optical-flow magnitude exceeds 0.15, producing 51K animations, 28K objects, 246K one-second clips, and 12M videos.The flow magnitude is averaged over videos rendered from four orthogonal fixed cameras.
  • Dataset and model: Each frame outputs 128 × 128 × 4 = 65,536 Gaussians, which are rendered at 512x512 resolution for supervision.The model uses a pretrained LGM backbone with added temporal attention layers.
  • Training: Training combines LPIPS and RGB MSE losses with segmentation-mask MSE supervision, using RGB and alpha rendering functions.The rendered image and mask are indexed by camera pose and timestep.
  • Training: Training uses 8-FPS videos with T = 8 on 128 80G A100 GPUs for 200 epochs, taking about one day.One eight-frame clip is trained per GPU at each iteration.
  • Inference: On an RTX 4080 Super, a forward pass takes about 0.3 seconds and interpolation takes 0.065 seconds between every two frames.A ten-second 30-FPS video can be reconstructed and interpolated to 45 FPS in 15 seconds.

D Full Quantitative Results on Consistent4D Benchmark

The Consistent4D appendix documents full benchmark metrics, training-data examples, camera-alignment processing, and a HexPlane ablation. It also provides the user-study instruction screenshot and related qualitative illustrations.

  • Benchmark and visual analysis: Table 4 reports the full metric comparison between L4GM and state-of-the-art approaches on the Consistent4D benchmark.The caption states that baseline results come from reference [15].
  • Benchmark and visual analysis: Training examples use masked input views that are replaced by copies of the multiview images at time t = 1.This illustrates how the model receives initial multiview information during training.
  • Benchmark and visual analysis: Azimuth alignment selects the rendered view minimizing image-space difference from the input frame, then constructs sparse views from that aligned angle and three orthogonal poses.The selected angle is θ_align = argmin_θ ||f(P_static, θ) − I_1||2.
  • Benchmark and visual analysis: The appendix includes a HexPlane ablation that predicts a canonical Gaussian and deformation field, producing T Gaussian sets through a decoder with cross-view and temporal attention.The decoder outputs six planes from the U-Net middle block.
  • Benchmark and visual analysis: Figure 10 shows the instructions given to participants for comparing L4GM with baselines.The figure is part of the user-study materials.

G.1 Evaluation Dataset

The evaluation uses 24 videos spanning generated and real-world sources, standardized preprocessing, qualitative comparisons, and human studies. Results include additional qualitative comparisons on the Consistent4D dataset.

  • Evaluation dataset: The evaluation dataset contains 24 videos from Emu, Sora, Veo, and ActivityNet, covering generated and real-world content.The sources contribute 10, 7, 5, and 2 videos respectively.
  • Evaluation dataset: Videos are foreground-segmented with SAM-Track, cropped and rescaled to 512x512, then standardized to 64 frames for comparison.Thirty-FPS videos are downsampled to 15 FPS and all videos are trimmed to four seconds.
  • User studies: Human evaluations compare L4GM with DG4D, OpenLRM, and STAG4D using generated 4D scenes rendered from similar camera perspectives.The studies assess five evaluation axes and include an equally-good option.
  • User studies: Each video pair is evaluated by five participants, totaling 120 responses for each baseline comparison.Participants are selected from English-speaking countries and must have high task acceptance rates and substantial platform experience.
  • Qualitative results: The appendix provides further qualitative comparisons on generated and real-world videos, plus additional Consistent4D results.Figures 11–13 contain these supplementary comparisons.

I Limitations

L4GM has difficulty with ambiguous motion, multiple mutually occluding objects, and ego-centric viewpoints. These limitations are illustrated through failure cases and qualitative results on real-world and Consistent4D videos.

  • Motion ambiguity can produce unnatural leg motion from views other than the reference, even when alignment with the reference view succeeds.The failure cases identify motion ambiguity as a limitation of reconstruction quality across viewpoints.
  • Multiple objects are reconstructed poorly, particularly when they occlude each other.
  • Ego-centric videos remain difficult because the model assumes input views at 0° elevation.This limitation is represented among the reported failure cases.
  • Additional qualitative comparisons are reported for real-world videos and the Consistent4D dataset.
Loading 2406.10324v1…