Racing in Volume with Flow Ensembles

• At 30 Hz, laterally filmed Formula 1 cars move 200–400 pixels between frames, beyond the reliable range of optical flow and point trackers.Source: §1 · p. 2
• Existing deformable methods degrade under large displacements, while streaming methods relying on one correspondence mechanism can propagate its failures into reconstruction.Source: §1 · p. 2
• Driving datasets lack fixed trackside lateral motion, and indoor benchmarks pair slow dynamics with controlled lighting, leaving the target setting underexplored.Source: §1 · p. 2
• Monaco4D is a multi-view outdoor benchmark with Formula 1 sequences, varied illumination, multiple camera modalities, and dense ground-truth annotations.Source: §1 · p. 2
• FastFlowGS fuses sparse matches, semi-dense tracks, and dense optical flow, using cross-level agreement to assess reliability before uncertainty-aware 3D lifting and Kalman-prior fusion.Source: §1 · p. 2
• On CMU-Panoptic, FastFlowGS reports 12.6% higher VMAF and 35% greater efficiency, and it is the first streaming reconstruction method evaluated on Monaco4D.Source: §1 · p. 2

• The related-work context identifies fast-moving outdoor subjects viewed from spectator positions as largely unexplored, and Monaco4D covers six circuit sections with dense multi-view annotations under five illumination conditions.Source: §2 · p. 2, Figure 2 · p. 3
• Deformable representations and Gaussian playback methods generally assume slow deformation or camera-dominated motion and degrade under rapid displacement.Source: §2 · p. 2
• Streaming methods use varied acceleration strategies, but TrackerSplat initializes Gaussians from point trackers, whereas FastFlowGS extends it with multi-resolution fusion, uncertainty-aware triangulation, and a variational position prior.Source: §2 · p. 2
• Outdoor reconstruction methods largely assume ego-motion and forward parallax, while sports systems focus on compact objects or controlled-lighting subjects moving up to 10 m/s.Source: §2 · p. 3

• Monaco4D contains six sequences rendered in Unreal Engine 5 on a to-scale Monaco circuit replica, with single- and multi-car variants and realistic maneuvers.Source: §3 · p. 3
• Each sequence includes fixed trackside views, vehicle-mounted onboard views, and drone trajectories simulating broadcast behaviors such as tracking, panning, and zooming.Source: §3 · p. 3
• Five illumination conditions—day, evening, night, fog, and rain—include within-clip shifts such as tunnel entries, building shade, and nighttime streetlamp pools.Source: §3 · p. 3
• Frames provide path-traced RGB, surface normals, metric depth, instance masks, and dense 3D scene flow as ground truth.Source: §3 · p. 3
• FastFlowGS separates sky, background, and foreground; it fuses triangulated multiresolution motion estimates with uncertainty and a Kalman temporal prior to initialize per-Gaussian positions.Source: Figure 3 · p. 4

4.1 Scene Factorization for Dynamic Reconstruction

• FastFlowGS represents scenes with static-background, dynamic-foreground, and sky layers, updating foreground geometry while freezing background geometry after initialization.The background’s spherical harmonics are updated for illumination drift; the sky is rendered as a low-resolution skybox.Source: §4.1 · p. 4

4.2 Multi-Resolution Motion Field Estimation

• Sparse keypoints, semi-dense point tracks, and dense per-pixel optical flow provide motion fields at different spatial scales.Sparse and medium tracks are densified with Gaussian-weighted Voronoi interpolation, whose distance confidence down-weights pixels far from observations.Source: §4.2 · p. 4

4.3 Cross-Level Motion Agreement

• Cross-level disagreement lowers a tracker’s confidence regardless of its density, while distance from observations also signals unreliable motion.The confidence terms encode complementary checks: Voronoi distance and disagreement between levels.Source: §4.3 · p. 5

4.4 Gaussian-to-Pixel Association

• Gaussian-to-pixel association weights triangulation by each Gaussian’s rendering contribution, tracker confidence, and motion magnitude.Top-K Gaussian indices and alpha-blending weights associate image observations with Gaussians; the projected means are then displaced using weighted flow.Source: §4.4 · p. 5

4.5 Uncertainty-Aware Multi-View Triangulation

• Multi-view triangulation estimates each Gaussian’s 3D position from weighted viewing rays and discards views with motion nearly collinear to the ray.Under isotropic observation noise, covariance measures geometric reliability: large eigenvalues indicate poor geometry, and that covariance becomes measurement uncertainty for temporal fusion.Source: §4.5 · pp. 5–6

4.6 Variational Temporal Motion Fusion

• A Kalman-style update combines tracker measurements according to triangulation precision with a temporal prior whose covariance grows across unanswered frames.The objective balances temporal deviations against disagreement with each tracker; when all levels fail to triangulate, only the temporal term contributes.Source: §4.6 · p. 6
• Process noise is tied to median 3D displacement among successfully triangulated Gaussians, relaxing the temporal prior in fast frames and tightening it in near-static frames.High-uncertainty Gaussians are split or cloned before optimization, placing capacity where multi-view geometry is weakest.Source: §4.6 · p. 6

5 Experiments

• The evaluation pairs CMU-Panoptic at native and 5×-subsampled frame rates with Monaco4D full-quality and reduced-memory protocols.The study uses six CMU-Panoptic subsequences and three Monaco4D sequences.Source: §5 · p. 7, §5.2 · pp. 8–9

5.1 CMU-Panoptic

• At both frame rates, FastFlowGS leads every baseline on every metric and keeps ∆-PSNR near zero while most baselines drift more than 1 dB.∆-PSNR measures the difference between first- and last-frame PSNR; higher values indicate greater sequence degradation.Source: §5 · p. 7, §5.1 · p. 8, Table 1 · p. 8
• Under 5× subsampling, FastFlowGS retains the highest PSNR and M-PSNR and lowest ∆-PSNR; FastFlowGS-faster has the highest VE at 3 seconds per frame.The 5× setting tests motion under larger inter-frame displacements.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 Monaco4D

• On Monaco4D, FastFlowGS outperforms 3DGS-Base on every sequence and metric, with gains largest on Fairmont and smallest on Rascasse.The three sequences span five mutually occluding cars at a hairpin, three cars at maximum speed, and a single car, respectively.Source: §5.2 · p. 8, Table 2 · p. 9
• Under the full-quality protocol, all five streaming baselines lose most Gaussians by frame 2 and produce empty reconstructions by frame 5.Without densification, displaced Gaussians lose their gradient signal; with densification, recovery resembles restarting 3DGS from scratch each frame.Source: §5.2 · pp. 8–9, Figure 7 · p. 10
• FastFlowGS achieves the highest VE on every Monaco4D sequence, despite operating against a background eight times denser than any baseline.The reduced-memory evaluation replaces the hierarchical background with a standard 3DGS at roughly 1 million Gaussians.Source: §5.2 · p. 9, Table 3 · p. 9

5.3 Ablations

• Sparse and dense correspondences alone reach similar CMU-Panoptic M-PSNR, while dense correspondences lead on Monaco4D; fusing both beats either alone on every metric.The ablation table reports iterations to reach 24+ PSNR on CMU-Panoptic and 18.5+ PSNR on Monaco4D.Source: §5.3 · p. 10, Table 4 · p. 11
• The Kalman temporal prior gives the largest component gain: VMAF rises 2.55 points on CMU-Panoptic and 2.76 on Monaco4D, while convergence drops from 260 to 200 and 760 to 700 iterations, respectively.The disagreement filter removes 40 iterations on CMU-Panoptic and 80 on Monaco4D; the authors attribute the prior’s gain to constraining Gaussian positions to smooth trajectories.Source: §5.3 · p. 10, Table 4 · p. 11

6 Conclusion

• FastFlowGS outperforms all streaming baselines on CMU-Panoptic and is the only method to produce coherent Monaco4D reconstructions, where existing approaches fail entirely.The paper identifies initialization quality, rather than optimization budget, as the bottleneck; fused correspondences and a Kalman temporal prior yield faster-converging, more stable initializations.Source: §6 · p. 11

7 Limitations and Failure Cases

• The motion model updates positions, not rotations or scales, so deforming or rapidly rotating objects must recover shape from scratch.This erodes the convergence advantage of good initialization.Source: §7 · p. 11
• Correspondence quality degrades on textureless surfaces, distant objects, and depth-dominant motion; disagreement scoring cannot fully compensate.Source: §7 · p. 11
• Tracker levels can share image evidence, making the fused covariance overconfident and reducing optimizer correction under tight iteration budgets.Source: §7 · p. 11
• Both FastFlowGS and Monaco4D assume a static background; simultaneous multi-view mask failures have no recovery path, and real outdoor F1-scale performance is uncharacterized.Moving spectators or trackside elements can produce localized ghosting; Monaco4D is synthetic.Source: §7 · p. 11

8.1 Summary of method

• The per-frame position initialization minimizes a quadratic cost combining a temporal dynamics term with tracker measurements weighted by triangulation covariance.Its precision-weighted solution fuses tracker estimates with a Kalman-style temporal prior and provides posterior covariance for downstream processing.Source: §8.1 · p. 15

8.2 Additional experiments with competing methods

• The low-memory experiments report PSNR and MPSNR; their captions describe comparable quality with faster convergence, while QUEEN runs out of memory on Main Straight.Source: §8.2 · p. 15, Table 6 · p. 17, Table 8 · p. 18
• Foreground-background separation lets competing methods attempt corrections over time instead of permanently losing dynamic Gaussians outside the segmentation mask.Source: §8.2 · p. 15
• MPSNR can rank competing methods higher because blur may better match average texture, while FastFlowGS produces sharper detail at the cost of sometimes noisier texture.The paper notes that a sharp edge misalignment can hurt MSE more than global blur.Source: §8.2 · p. 15
• FastFlowGS is typically best or second-best across metrics, reconstructing scenes with more Gaussians, faster, and at better quality.Source: §8.2 · p. 15

8.3 Timing breakdown of FastFlowGS

• FastFlowGS cuts sequential finetuning from 738 s to 9.5 s/frame (78×); TrackerSplat takes 72 s with the same point tracker.Within the dynamic lane, initialization adds less than 1 s and finetuning accounts for about 90% of lane cost; preprocessing is excluded from reported timing comparisons.Source: §8.3 · p. 15

8.4 Confidence-Weighted Voronoi Interpolation

• Gaussian-weighted K-nearest-neighbor interpolation densifies sparse and medium tracks while producing target visibility and per-pixel distance confidence.That confidence downweights pixels far from observed tracks through agreement scoring and triangulation; experiments use K = 8 and σ = 0.05.Source: §8.4 · p. 20

9 Monaco4D: Additional Details

• Monaco4D sequences use Unreal Engine 5 path-traced rendering at real-world scale, with 1 Unreal Unit equal to 1 cm.Source: §9 · p. 21

9.1 Camera Configuration

• Cameras share fixed 16:9, 12 mm, f/2.8 intrinsics; viewpoints include trackside, seven onboard positions per vehicle, and three drone trajectories.At t = 0, dense hemispherical cameras bootstrap each vehicle's first-frame reconstruction, but those views are excluded from evaluation.Source: §9.1 · p. 21, Figure 9 · p. 21

9.2 Environment and Track Geometry

• Track geometry and camera placements are metric, but surrounding city structures prioritize visual realism and occlusion fidelity over survey-grade geometric accuracy.Source: §9.2 · p. 21

9.3 Weather and Illumination Layers

• Separate daytime, night, fog, and rain layers change environmental appearance while preserving scene geometry, camera setup, and layout.Each layer stores its sky, lighting, atmospheric, cloud, and weather settings.Source: §9.3 · pp. 21–22
• Seven onboard camera positions per car follow FIA broadcast regulations.The figure distinguishes forward- and rear-facing viewpoints.Source: Figure 10 · p. 22

9.4 Vehicle Models and Animation

• Vehicles use physics-based suspension, circuit-positioned DRS actuation, steering-linked driver hands, and suspension-driven head motion rather than canned animation.These rigging mechanisms make vehicle appearance respond frame to frame to track forces.Source: §9.4 · p. 22
• Peak inter-frame displacements reach 200–400 pixels for laterally oriented trackside cameras, beyond standard optical-flow networks’ operating range.Simulated lap-segment durations fall within 90% of real-world counterparts; speeds were reduced roughly 10% to increase frames captured per pass.Source: §9.4 · p. 22

9.5 Rendering Configuration

• Table 9 reports camera and frame counts per sequence and variation, plus the average number of views containing at least one car per frame.The rendering setup uses a 128-frame warm-up before each sequence so lighting and temporal state can converge.Source: Table 9 · p. 22, §9.5 · p. 23
• Frames use Unreal Engine Path Tracer rendering with 64 spatial samples per pixel, one temporal sample, and a 2-frame denoising window.Frames are exported as 8-bit PNGs at 150% screen percentage, with instanced-mesh geometry culling disabled.Source: §9.5 · pp. 22–23
• Vehicle rigs include suspension-driven body movement and head bobbing, circuit-triggered DRS actuation, and steering-linked driver hands.These mechanisms make appearance respond frame to frame to track forces rather than canned animation.Source: Figure 11 · p. 23

9.6 Lighting and Atmosphere

• Illumination combines directional sunlight, diffuse skylight, Rayleigh and Mie atmospheric scattering, and volumetric clouds.The cloud layer adds natural skylight variation and soft shadowing.Source: §9.6 · p. 23

9.7 Post-Processing and Motion Blur

• Motion blur is disabled, and no tone mapping, stylization, or image-space effects are applied.A global Post Process Volume enables the ray-tracing features needed by the path tracer.Source: §9.7 · p. 23

9.8 Frame Rate

• All sequences are rendered at 30 FPS, with selected sequences additionally rendered at 240 FPS for denser temporal sampling.The higher-rate sequences support tracking, reconstruction, and temporal interpolation tasks.Source: §9.8 · p. 23
• The circuit figure marks seven sequence segments, trackside camera placements, and drone trajectories.Its insets show the density and angular spread of trackside coverage.Source: Figure 12 · p. 24