Source-linked AI summary
FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation
Orest Kupyn, Goutam Bhat, Philipp Henzler, Fabian Manhardt, Christian Rupprecht, Federico Tombari
TL;DR
Generating explorable 3D scenes from one image requires geometrically accurate surface representations, while existing latent decoders mainly produce volumetric Gaussians without well-defined surfaces. FLAT directly decodes triangle splats from video diffusion latents in one pass, achieving better geometric accuracy with competitive visual quality and enabling opaque, real-time-renderable representations after lightweight refinement.
Problem
Existing feedforward latent scene decoders produce volumetric Gaussians rather than well-defined opaque surfaces needed by standard graphics engines.
Method
FLAT decodes semi-opaque triangle-splatting primitives directly from frozen video diffusion latents using ray-centered parameterization and a modified product window function.
Results
FLAT achieves significantly better geometric accuracy with competitive visual quality and converts predicted triangles into a more coherent opaque mesh through lightweight post-processing.
Takeaways & Limitations
FLAT demonstrates that explicit non-volumetric surface primitives can be decoded from video latents for practical feedforward scene generation and real-time rendering.
Takeaways & Limitations
Triangle representations struggle with thin structures, reflections, transparency, fine details, and pixel-level quality, while converted meshes may remain sparse rather than watertight.
Abstract
from arXiv · showhide
Generating explorable 3D scenes from a single image requires strong generative priors and accurate geometric representations suitable for downstream use. Current video diffusion models offer high-quality generation and implicitly encode multi-view geometric structure in latent space. However, existing feedforward latent scene decoders typically output volumetric 3D Gaussians that lack a well-defined surface, limiting their use in simulation or standard graphics pipelines. This motivates decoding surface-aligned primitives that are not only renderable but also closer to explicit geometric assets. We ask whether compressed video diffusion latents can be mapped directly to explicit surface primitives in a single pass. To this end, we introduce FLAT and, for the first time, show that triangle splats can be decoded directly from video diffusion latents. Compared with decoding 3D Gaussians, predicting flat primitives is notoriously more challenging due to high sensitivity to primitive orientations, oftentimes leading to poor gradient flow. FLAT solves with two key ingredients: a ray-centered rotation parameterization for triangle regression and a novel product window function that improves gradient flow during differentiable triangle rendering. On standard benchmarks, FLAT achieves significantly better geometric accuracy while maintaining competitive visual quality compared to state-of-the-art feedforward baselines. We further show that a lightweight test-time refinement step converts the predicted triangle soup into a fully opaque, game-engine-ready representation that supports real-time rendering. By evaluating 3DGS, 2DGS, and triangle splatting variants under an identical training setup, we provide the first systematic analysis of representation tradeoffs in feedforward scene generation. The project page is available at https://flat-splat.github.io
1 Introduction
FLAT addresses the need for geometrically accurate, physically grounded, explorable 3D scenes by directly decoding explicit triangle-splatting surfaces from frozen video diffusion latents in one pass. It targets the optimization and gradient-flow difficulties that make feedforward flat-primitive prediction harder than volumetric Gaussian decoding.
- Motivation: Explorable 3D environments require visually plausible, geometrically accurate, physically grounded representations for novel views, simulation, and interaction.The required representation must capture 3D layout, surface structure, and scale.
- Limitations: Existing latent scene decoders generate only volumetric 3D Gaussians, whose semi-transparent blob geometry limits their suitability as explicit surface assets.Video-latent decoders can produce high-quality explorable scenes, but their output remains restricted to 3D Gaussians.
- Challenges: Feedforward triangle decoding is difficult because opaque rendering is non-differentiable, vertex regression can become degenerate, and triangle quality is highly sensitive to orientation.These challenges lead to poor gradient flow compared with volumetric Gaussians.
- FLAT: FLAT directly predicts semi-opaque triangle-splatting primitives from frozen video diffusion latents in a single forward pass, using depth and normal supervision for geometric accuracy.The model is designed to produce a physically grounded scene representation from an input image.
- FLAT: FLAT introduces a ray-centered local triangle parameterization, constrained Cholesky-style shape, and residual orientation prediction around a ray-aligned frame.These ingredients make feedforward flat-primitive decoding practical.
2 Related Work
Related work spans generate-then-optimize and geometry-free novel-view synthesis, latent-space feedforward scene decoders, and explicit 3D representations. FLAT extends this line by directly predicting triangle splats from video latents to improve geometric accuracy and rendering compatibility.
- Novel View Synthesis and Scene Generation: Early methods expand a multiview set before reconstructing an explicit 3D representation, while ViewCrafter applies generate-then-optimize to video diffusion.ViewCrafter generates large, dense views using a point-cloud-conditioned video model.
- Novel View Synthesis and Scene Generation: Geometry-free methods omit explicit scene parameters, with LVSM mapping input images directly to novel views and Genie targeting near-real-time generation with high 3D consistency.LagerNVS further uses features from geometry foundation models to support geometry-free generation.
- Novel View Synthesis and Scene Generation: Wonderland decodes 3D Gaussian Splatting scenes directly from video diffusion latents, but must infer geometry, appearance, and depth from rendering losses in compressed latent space.The decoder is also guided by often imperfect cameras, while Generative Gaussian Splatting achieves its highest quality after second-stage scene optimization.
- 3D Scene Representations: NeRF-style volumetric representations enabled view synthesis, whereas 3D Gaussian Splatting improved efficiency and real-time rendering but may lack surface regularity and geometric precision.The passage characterizes Gaussian representations as flexible and efficient while noting limitations for explicit surface structure.
- 3D Scene Representations: FLAT addresses the difficulty of feedforward non-volumetric scene generation by predicting triangles directly from video latents despite compact gradients and precise orientation requirements.This design supports efficient, flexible scene generation with strong geometric accuracy and compatibility with modern rendering engines.
3 Method
FLAT maps frozen video-diffusion latents and camera trajectories from a single RGB image directly to a world-space scene of triangle splats in one forward pass. Its ray-centered parameterization and product-style window function stabilize triangle regression and differentiable rendering.
- Feedforward scene decoding: FLAT conditions a feedforward scene decoder on a single RGB image and target camera trajectory, mapping denoised video latents directly to world-space surface primitives.The frozen video generator supplies the latent representation, while the decoder predicts explicit scene parameters for arbitrary-view rendering.
- Triangle splat representation: Each triangle splat comprises three 3D vertices, color, smoothness, and opacity, and is rendered through projected soft coverage with front-to-back alpha compositing.The primitives are projected with a pinhole camera model before overlapping contributions are accumulated in depth order.
- Triangle parameterization: FLAT predicts triangles in a ray-centered local frame, using a canonical equilateral triangle and lower-triangular transform to guarantee positive area and avoid degeneracy.The transform’s positive diagonal terms preserve valid triangles, while its off-diagonal term controls shear.
- Triangle parameterization: Residual tilt and spin angles around a ray-tangent frame provide more stable orientation regression than direct world-space rotations such as quaternions.Direct world-space rotation can cause unstable orientation, vanishing render support, and model divergence early in training.
- Differentiable rendering: The window function extends triangle support beyond its exact boundary and routes gradients through the full triangle rather than a max-selected edge.Compared with the original triangle-splatting formulation, avoiding the max reduction improves gradient flow in the feedforward latent model.
- Architecture and training: The decoder reuses Wan-2.1’s pretrained RGB decoder backbone, adds zero-convolution camera conditioning and triangle-parameter heads, and trains only the scene decoder from frozen video latents.Training uses videos with known camera trajectories, depth, and normal maps; inference can decode larger scenes despite training on shorter sequences.
4 Evaluation
FLAT is evaluated for single-image feedforward 3D scene generation on RealEstate10K and DL3DV using matched representation variants and standard comparison protocols. The evaluation shows that triangle splatting improves geometric accuracy while preserving strong visual quality and supports lightweight opaque-mesh conversion for downstream graphics.
- Evaluation setup: FLAT is evaluated on RealEstate10K and DL3DV alongside comparable 3DGS and 2DGS variants trained with identical hyperparameters and protocols.State-of-the-art 3DGS methods are also included for comparison.
- Results: Triangle splatting produces significantly more accurate geometry than other representations while maintaining high visual quality relative to state-of-the-art methods.The 3DGS variant achieves the highest visual fidelity, supporting the effectiveness of the training pipeline and design choices.
- Geometric evaluation: Geometric accuracy is measured by comparing rendered normal maps with ground-truth normals, using Metric3D-v2 for FLAT and finite differences for the 3DGS variant.The evaluation reduces model bias because FLAT is supervised with NormalCrafter.
- Representation analysis: Triangles recover sharper, more geometrically faithful surfaces and provide an explicit non-volumetric representation better aligned with mesh extraction and real-time graphics pipelines.These results support explicit triangle-based feedforward decoding when geometric accuracy and downstream compatibility matter.
- Opaque mesh conversion: FLAT predictions can be converted into an opaque mesh with a lightweight post-processing step, which is compared against mesh conversion from 2DGS and 3DGS.Existing 2DGS and 3DGS surface-extraction methods rely on dense view coverage and are sensitive to hyperparameter choices.
- Ablations: Ablations examine parameterization, rendering, conditioning, post-processing, and architecture, showing that stable triangle decoding depends on combining the main FLAT components.The studied choices include ray-centered triangle parameterization, the modified triangle window function, rotation parameterization, and model architecture.
5 Conclusion
FLAT generates 3D scenes from a single image by combining a frozen camera-conditioned latent video model with a lightweight decoder that predicts triangle splats in one forward pass. Its surface-oriented parameterization and differentiable rendering formulation avoid per-scene optimization while supporting plausible novel views and real-time rendering.
- Method: FLAT combines a frozen camera-conditioned latent video model with a lightweight decoder that directly predicts triangle splats in a single forward pass.The method is designed for 3D scene generation from one input image.
- Capabilities: The design avoids expensive per-scene optimization while enabling plausible generation beyond the input view and real-time rendering.
- Representation: FLAT demonstrates that non-volumetric surface primitives can be decoded from video latents when parameterization and rendering are chosen carefully.The approach uses a ray-centered triangle parameterization and a modified differentiable triangle-splatting formulation to improve optimization.
A Pipeline Flexibility
FLAT can replace Wan-2.1’s standard RGB decoder and map denoised latents directly to explicit triangle-based scene geometry without modifying or finetuning the upstream video generator. Because Wan variants share the latent space, the same decoder transfers across image-to-video, text-to-video, video-to-video, interactive, and world-consistent pipelines.
- Decoder-swap design: FLAT attaches to denoised Wan-2.1 latents without modifying the latent space or finetuning the diffusion transformer.At inference, it can simply be added as a decoder or replace the standard VAE RGB decoder.
- Decoder-swap design: Any Wan-2.1 variant finetuned from the base model can produce explicit triangle-based scene geometry instead of RGB frames.The supplied passage identifies image-to-video and other pipeline variants as examples, though its list is truncated.
- Decoder-swap design: A shared latent representation lets FLAT reuse one scene decoder across pipeline modes rather than requiring a separate 3D decoder for each mode.This design allows upstream improvements in motion quality, conditioning, or control interfaces to transfer without retraining a separate scene decoder for every variant.
- Pipeline coverage: The same latent scene decoder can attach to image-to-video, text-to-video, video-to-video, interactive, and world-consistent Wan pipelines.Figure 5 describes FLAT as replacing the standard RGB decoder because these Wan variants share the same latent space.
- Pipeline coverage: Attaching FLAT to a Wan-2.1 text-to-video pipeline converts text-to-video latents into explicit geometry with rendered views and predicted normal maps.Figure 6 presents examples using rendered views alongside corresponding predicted normal maps.
B Post Optimization
FLAT’s optional short test-time optimization improves visual and geometric quality by correcting common feedforward failures. Aggressive pruning further cleans geometry by removing weak triangles that blur surface orientations and normal boundaries.
- Optimization: A short test-time refinement improves both visual and geometric quality by correcting common failures in the feedforward prediction.The feedforward output provides a strong initialization, allowing refinement to focus on failure cases.
- Optimization: Refinement can require as few as 250 steps.The passage describes this as a very short optimization pass.
- Optimization: The optimization addresses surface misalignment, semi-transparent structures, floating low-importance triangles, thin objects, and overly diffuse normal predictions.
- Pruning: Aggressive pruning removes weak or unsupported triangles, producing sharper normal maps and cleaner local geometry.Diffuse low-opacity triangles can hide RGB-space errors while blurring surface orientation and softening normal boundaries.
- Evaluation: The post-optimization pass consistently improves the feedforward prediction on RealEstate10K.
C Limitations and Broader Impact
FLAT’s triangle representation and feedforward generation remain limited by imperfect mesh connectivity, sparse geometry, challenging fine details, and constrained training scale. The approach offers practical benefits for geometry-sensitive applications but also raises misuse and computational-impact concerns.
- Limitations: Triangle splats improve surface alignment but are not optimized for standard novel-view synthesis performance.Their explicit, non-volumetric representation creates a tradeoff between geometric structure and synthesis quality.
- Limitations: Opaque-mesh conversion improves usability, but the resulting geometry remains sparse, fragmented, and incomplete without additional post-processing.A clean, densely connected, watertight mesh remains an open problem for scene mesh recovery methods.
- Broader Impact: FLAT could support simulation, robotics, gaming, and AR/VR, while also enabling realistic synthetic environments or deceptive media.Training and deployment additionally require substantial computation, and improving realism can lower the cost of misleading content.
- Limitations: Thin, elongated surfaces, tiny details, and reflections remain challenging to model with triangles.Triangles may generate sharper details while achieving lower PSNR than Gaussians, which are optimized directly for PSNR.
D Training Details
FLAT uses a four-stage training pipeline with progressive resolution and view scaling, beginning with rapid decoder adaptation on 17-view RealEstate10K sequences. A lightweight conversion step turns semi-opaque predicted triangles into an opaque, game-engine-compatible mesh while preserving high scene-render quality.
- Progressive Training Schedule: The four-stage training pipeline progressively increases resolution and view scaling across training stages.Stage 1 uses 20,000 iterations at 320p, while Stage 2 uses 40,000 iterations at 320p.
- Progressive Training Schedule: Stage 1 trains on 17-view RealEstate10K sequences with 17 input-conditioning views and 17 target views to quickly adapt the decoder.Stage 1 runs for 20,000 iterations at 320p resolution.
- Progressive Training Schedule: Stage 2 trains on 49-view trajectories from mixed real and synthetic videos, using 49 sampled target views for supervision.Stage 2 runs for 40,000 iterations at 320p resolution.
- Mesh Conversion: The lightweight conversion step transforms semi-opaque predicted triangles into an opaque, game-engine-compatible mesh while retaining high scene-render quality.Accurate semi-opaque initial predictions simplify the conversion process.
E Scene Decoder Architecture
The scene decoder follows the Wan-2.1 VAE architecture, combining video latents with Plücker embeddings through a lightweight adapter before processing them with a causal 3D UNet-style decoder. Its design preserves temporal causality, stabilizes initialization, and uses staged upsampling to produce the final output tensor.
- Input Fusion and Causality: The decoder uses CausalConv3D with padding only on past frames to maintain temporal causality.Its input video latents xv ∈ R^B×16×T′×H′×W′ and Plücker embeddings xp ∈ R^B×32×T′×H′×W′ are fused by a lightweight adapter.
- Input Fusion and Causality: The adapter’s final projection is zero-initialized so geometry conditioning does not destabilize pretrained latents at initialization.
- Decoder Structure: The fused latent is processed by a 3D UNet-style decoder using RMSNorm, SiLU activations, and Scaled Dot-Product Attention throughout.Residual blocks comprise RMSNorm, SiLU, and CausalConv3D layers.
- Upsampling: Temporal upsampling occurs only in the first two upsampling stages, producing a fixed 4× temporal expansion.The final upsampling stage defaults to an identity pass.
- Upsampling: The decoder’s output tensor is 2× strided relative to the original image dimensions.
F Mesh Conversion Analysis
Direct conversion from soft triangles produces well-formed meshes with almost no degenerate faces or isolated triangles. The resulting connectivity averages 3.1 neighboring triangles per face, near the value expected for regular manifold surfaces.
- 0.02% of faces are degenerate, indicating highly well-formed local geometry after direct conversion from soft triangles.
- 0.00% of triangles are fully isolated or disconnected in the converted meshes.
- 3.1 neighboring triangles connect to each triangle on average, close to the expected value of 3 for regular manifold surfaces.