Source-linked AI summary
WildSmoke: Ready-to-Use Dynamic 3D Smoke Assets from a Single Video in the Wild
Yuqiu Liu, Jialin Song, Manolis Savva, Wuyang Chen
TL;DR
Most existing fluid reconstruction approaches rely on controlled multi-view recordings, leaving smoke captured in the wild underexplored. WildSmoke reconstructs dynamic 3D smoke assets from a single video using smoke extraction, pose and particle initialization, and inferred multi-view supervision. It reports a +2.22 dB average PSNR improvement on wild videos and supports editing through fluid simulation.
Problem
Existing fluid reconstructions rely heavily on controlled multi-view recordings, while single-camera wild videos have noisy backgrounds, unknown poses, and coupled viewpoints and timesteps.
Source: §1 · p. 1, §2.2 · p. 2Method
The pipeline extracts smoke, initializes particles and camera poses, and decouples spatial and temporal trajectories using generated multi-view videos and local pose perturbation.
Source: §1 · p. 1, §3.4 · p. 4Results
+2.22 dB average PSNR improvement over prior work across diverse real-world videos with light and dense smoke.
Source: §4.3 · p. 7Takeaways & Limitations
The reconstructed assets support physically consistent smoke editing through fluid simulation for downstream simulation workflows.
Source: §6 · p. 8The setup keeps smoke centered and uses 7° rotation during future steps, limiting its demonstrated scope to centered smoke and modest future camera motion.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
We propose a pipeline to extract and reconstruct dynamic 3D smoke assets from a single in-the-wild video, and further integrate interactive simulation for smoke design and editing. Recent developments in 3D vision have significantly improved reconstructing and rendering fluid dynamics, supporting realistic and temporally consistent view synthesis. However, current fluid reconstructions rely heavily on carefully controlled clean lab environments, whereas real-world videos captured in the wild are largely underexplored. We pinpoint three key challenges of reconstructing smoke in real-world videos and design targeted techniques, including smoke extraction with background removal, initialization of smoke particles and camera poses, and inferring multi-view videos. Our method not only outperforms previous reconstruction and generation methods with high-quality smoke reconstructions (+2.22 average PSNR on wild videos), but also enables diverse and realistic editing of fluid dynamics by simulating our smoke assets. We provide our models, data, and 4D smoke assets at [https://autumnyq.github.io/WildSmoke](https://autumnyq.github.io/WildSmoke).
1. Introduction
Reconstructing smoke from in-the-wild video remains underexplored because existing approaches rely on controlled multi-view recordings. The paper proposes a pipeline for extracting and reconstructing dynamic 3D smoke assets from one video, with simulation-based editing.
1. Introduction
- Recovering velocity and density across the full spatiotemporal domain from visual inputs is the task of reconstructing fluid dynamics from imagery.The paper notes applications in cinematic smoke rendering and industrial fluid diagnostics.Source: §1 · p. 1
- Existing fluid reconstructions rely heavily on controlled multi-view recordings, while real-world smoke assets remain largely underexplored.Wild footage often has a moving camera and cluttered backgrounds, unlike recordings made with fixed, calibrated cameras and controlled smoke.Source: §1 · p. 1, §5.1 · p. 8
- The pipeline targets noisy backgrounds, unknown camera poses, and coupled camera viewpoints and timesteps with smoke extraction, pose and particle initialization, and inferred multi-view videos.Smoke extraction combines in-context segmentation with dehazing for light-smoke background removal.Source: §1 · p. 1
- +2.22 dB average PSNR improvement on in-the-wild videos over current fluid reconstruction or 3D generation methods.The reconstructed assets also support realistic smoke editing through interactive fluid simulation.Source: §1 · p. 2
2. Background
The background describes smoke as a dynamic 3D field represented by physical and visual particles. In the wild, reconstruction must contend with contaminated backgrounds, unknown camera poses, and entangled viewpoints and timesteps.
2.1. Smoke Reconstruction from Videos
- A smoke field is represented by physical particles for position and velocity, and grayscale visual particles with position, color, scale, opacity, and rotation attributes.Physical particles can map to density and velocity fields on grids through kernel-weighted interpolation.Source: §2.1 · p. 2
2.2. Smoke Reconstruction in the Wild: Challenges
- Wild-video reconstruction faces contaminated backgrounds, unmeasured camera poses, and a single camera trajectory that entangles spatial viewpoints with temporal frames.The passage describes an approximately one-to-one correspondence between camera viewpoints and timesteps.Source: §2.2 · p. 2
3. Methods
The method extracts clean smoke, initializes geometry and camera poses, adds multi-view supervision, trains particle representations, and prepares 4D smoke assets for simulation.
3.1. Smoke Extraction
- One-shot SAM annotation and SegGPT propagation yield smoke masks, which are smoothed; light smoke additionally uses fine-tuned DehazeFormer background recovery, while dense smoke skips dehazing.The method categorizes smoke by background visibility: dehazing is needed when background remains visible through light smoke.Source: §3.1 · pp. 2–3
3.2. Pose Estimation and Coarse Geometry
- DUSt3R initializes camera intrinsics, extrinsics, and focal lengths; its smoke-mask-filtered point cloud initializes physical and visual particles with spatial priors that support convergence.The unfiltered point cloud includes smoke and background regions, so the extracted masks retain only smoke-relevant points.Source: §3.2 · p. 3
3.3. Inferring Multi-View Videos
- SV4D 2.0 novel-view synthesis addresses coupled viewpoint-timestep trajectories; overlapping clip frames condition segments, and later generated frames receive exponentially decaying training weights.Novel viewpoints use azimuth offsets of −10°, 10°, 20°, and 30° relative to the current frame.Source: §3.3 · pp. 3–4
3.4. Training Gaussian Particles
- Visual and physical particles are optimized against rendered-view photometric error, with physical particles regularized by incompressible PBF; local pose perturbation adds nearby viewpoints as supervision.The perturbation shifts camera pose along the video trajectory by a small interval, progressively increased from 2 to 4 during training.Source: §3.4 · p. 4, Figure 4 · p. 4
3.5. Ready-to-Use 4D Smoke Assets
- Visual particles provide density and physical particles provide velocity for grid initialization; PhiFlow advances the flow with MacCormack advection and a pressure-Poisson solver, enabling wind or obstacle interactions.Visual and physical splatting use the same bounding region defined by the visual particles.Source: §3.5 · pp. 4–5
4. Experiments
Across real-world smoke videos, the method outperforms prior work; synthetic evaluations compare reconstruction quality across views and time, and reconstructed assets are also tested in interactive simulations.
4.1. Settings
- Evaluation uses synthetic paired videos, FLAME drone footage, and three Pixabay smoke videos; PSNR is averaged over frames, with SSIM and LPIPS reported in the appendix.Novel-view synthesis is evaluated only on synthetic data because real-world videos lack ground-truth multiview footage; real-world evaluation covers input-view future prediction only.Source: §4.1 · p. 5, §4.3 · p. 7
- Videos are standardized to 270 frames, with 240 training frames; synthetic cameras rotate 53° during training versus 7° in future steps.The novel view fixes the camera at t = 1, while the input view follows the source-video camera trajectory.Source: §4.1 · p. 5
4.2. Synthetic Data
- On synthetic data, the pipeline achieves higher PSNR and more stable visualizations than HyFluid and FluidNexus; Trellis cannot perform future prediction and its novel-view smoke fades to black by time T.Ablations show progressive gains from segmentation, pose/particle initialization, and multi-view supervision; adding multi-views raises novel-view future-prediction PSNR from 23.59 to 25.04 (+1.45).Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
4.3. Smoke Videos in the Wild
- +2.22 dB average PSNR improvement over prior work across light and dense smoke videos, with consistent reconstructions blending into their backgrounds.Real-world evaluation covers FLAME and Pixabay videos.Source: §4.3 · p. 7, Table 3 · p. 7, Table 4 · p. 7
4.4. Interactive Simulations
- The reconstructed “city” smoke asset supports interactive simulations with global or localized wind and a rigid spherical obstacle.The wind force is (0.005, 0, 0); the local wind acts within radius 30, and the obstacle has radius 10.Source: §4.4 · p. 7, Figure 8 · p. 8
4.5. Training and Inference Cost
- Across five in-the-wild videos, reconstruction takes 4.47 GPU-hours on average; DUSt3R-based pose initialization reduces Gaussian-particle training by 1.5 GPU-hours.Table 5 reports video resolutions and per-stage training and inference costs.Source: §4.5 · p. 8, Table 5 · p. 8
5. Related Works
Related work spans fluid reconstruction from video, physics-based simulation and editing, and generative methods for view-consistent dynamic scenes. WildSmoke draws on these directions to reconstruct smoke from unconstrained footage and support simulation-based editing.
5.1. Fluid Field Extraction from Videos
- Prior fluid-reconstruction methods include PDE-coupled NeRFs, particle-driven neural rendering, hybrid neural fields, and single-video reconstruction with generative priors; in-the-wild reconstruction into ready-to-use dynamic smoke assets remains underexplored.FluidNexus’s multi-view generator was fine-tuned on laboratory smoke with fixed calibrated cameras and clean backgrounds.Source: §5.1 · p. 8
5.2. Physics-based Fluid Simulation and Editing
- Building on simulation and differentiable-physics tools, WildSmoke reconstructs smoke from in-the-wild video and integrates simulation for editing in unconstrained real-world scenarios.Earlier production solvers enabled editing through external forces, boundary conditions, and solid–fluid interactions.Source: §5.2 · p. 8
5.3. 4D Generation
- WildSmoke uses view-consistent generative models to obtain auxiliary multiview trajectories, then reconstructs physically plausible dynamic smoke fields.Prior work improved view-consistent generation for images, novel views, and dynamic scenes.Source: §5.3 · p. 8
6. Conclusion
The conclusion presents a pipeline for recovering time-varying 3D smoke fields from a single in-the-wild video and producing dynamic assets. It reports that these assets support physically consistent editing through fluid simulation.
6. Conclusion
- The pipeline addresses noisy backgrounds, unknown camera poses, and coupled spatiotemporal views through segmentation and dehazing, pose estimation, and decoupled trajectories using multiview generation and local pose perturbation.The reconstructed assets support fluid-simulation editing and downstream simulation workflows.Source: §6 · p. 8
A.1. Synthetic Data
The synthetic evaluation places a 4D smoke sequence in a static 3D scene and renders it along a prescribed camera path. The appendix describes pose conversion, particle initialization, generated-view handling, and additional training choices.
A.1. Synthetic Data
- The synthetic sequence is rendered in a static 3D scene along a prescribed camera path, with training azimuth swept from −105° to −45° at a fixed 10° pitch.For novel-view testing, the camera is fixed to the first-frame pose.Source: §A.1 · p. 11, Figure 9 · p. 11
A.2. Pose Estimation
- DUSt3R estimates per-frame poses using the first frame as reference, and its camera-to-world poses are converted to the Gaussian Splatting convention by negating the y and z axes.Figure 10 visualizes the predicted poses and foreground point cloud for the valley video.Source: §A.2 · p. 11, Figure 10 · p. 11
A.3. Particle Initialization
- Physical and visual particles are initialized from foreground points selected by intersecting smoke segmentation with DUSt3R confidence masks, then axis-converted and voxel-downsampled.The method retains 100–300 foreground points per video.Source: §A.3 · pp. 11–12
A.4. Generated Videos
- Generated-video poses apply yaw and pitch offsets around the DUSt3R point-cloud center, which serves as the fixed look-at target.The world-space rotation is defined as ΔRt = Ry(Δϕt) Rx(Δθt).Source: §A.4 · p. 12
A.5. Down-Weighting of Generated Frames
- Generated-frame losses are exponentially down-weighted during visual-particle training to reduce the effect of their unreliability over time.The stated parameters are t0 = 0, k = 0.02, and wmin = 0.0.Source: §A.5 · p. 12
A.6. Learnable Buoyancy Strength
- The buoyancy coefficient is learnable and optimized jointly with reconstruction because buoyancy strength is underdetermined in wild footage.The paper contrasts this with well-controlled scenes, where relative buoyancy and gravity strength is fixed.Source: §A.6 · p. 12
A.7. High-Frequency Information Loss
- A frequency-domain loss compares FFT amplitude and phase on original input frames only; generated views are excluded because their high-frequency content is unreliable.The loss averages absolute differences across frequency bins and RGB channels.Source: §A.7 · p. 12
B.1. Smoke Extraction
The appendix shows smoke extraction examples, reports better SSIM and LPIPS for the method, and visualizes stable novel-view reconstructions over time. It also notes structural collapse in later SV4D 2.0 generated frames.
B.1. Smoke Extraction
- For light smoke, dehazing removes background leakage; for dense smoke, the masked result is used directly as the clean foreground.Figure 11 shows representative smoke extraction results across in-the-wild videos.Source: §B.1 · p. 12, Figure 11 · p. 12
B.2. Generated Videos from SV4D 2.0
- Later frames in SV4D 2.0 multi-view generation exhibit structural collapse.Figure 12 visualizes generated views over time.Source: Figure 12 · p. 13
B.3. SSIM and LPIPS
- The method performs better on both SSIM and LPIPS in smoke reconstruction comparisons on FLAME and Pixabay videos.Table 7 reports SSIM, where higher is better; Table 8 reports LPIPS, where smaller is better.Source: §B.3 · p. 13, Table 7 · p. 13, Table 8 · p. 13
B.4. Visualizations of Novel View Synthesis on Wild Videos
- Novel-view reconstructions keep smoke structure stable over time with the camera fixed at the first training-frame pose.Figure 13 shows results for several wild videos.Source: §B.4 · p. 13, Figure 13 · p. 13
B.5. Visualizations for Ablation Study on Synthetic Data
The appendix provides qualitative ablations alongside the main paper’s quantitative table, covering novel-view synthesis and future prediction on synthetic smoke videos.
B.5. Visualizations for Ablation Study on Synthetic Data
- Qualitative ablations in Figure 14 complement the quantitative ablation table in the main paper.The figure concerns novel-view synthesis and future predictions on synthetic smoke videos.Source: §B.5 · p. 13, Figure 14 · p. 14
- Figure 14 compares novel views from the camera pose at t = 1 with input views rendered along the source-video camera poses.GT denotes Ground Truth.Source: Figure 14 · p. 14
- The ablation trains on T = 240 frames and evaluates unseen future frames through T′ = 270.The figure includes future predictions beyond the training interval.Source: Figure 14 · p. 14