PaperlayerView exact revision on arXiv

Source-linked AI summary

Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platform

Yuning Gong, Yifei Liu, Yifan Zhan, Muyao Niu, Xueying Li, Yuanjun Liao, Jiaming Chen, Yuanyuan Gao, Jiaqi Chen, Minming Chen, Li Zhou, Yuning Zhang, Wei Wang, Xiaoqing Hou, Huaxi Huang, Shixiang Tang, Le Ma, Dingwen Zhang, Xue Yang, Junchi Yan, Yanchi Zhang, Yinqiang Zheng, Xiao Sun, Zhihang Zhong

arXiv:2512.08478v1cs.CVcs.AIcs.GR

TL;DR

Existing neural-rendering viewers are fragmented, deployment-heavy, or constrained in their support for dynamic content and generative models. Visionary combines browser-side WebGPU rendering with per-frame ONNX inference and a plug-in contract; it reports real-time performance and improvements over existing web renderers, while browser constraints still limit some models and post-processing.

  • Problem

    Existing viewer solutions impose deployment friction or legacy-pipeline constraints, limiting support for dynamic content and generative models across an expanding 3DGS ecosystem.

    Source: §1 · p. 2

  • Method

    Visionary couples WebGPU rendering with per-frame ONNX inference through a Gaussian Generator contract for browser-based, plug-in integration of 3DGS methods.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    Across representative static and dynamic scenes, Visionary sustains real-time interactive performance and improves frame time, dynamic update latency, and scalability over existing web-based renderers.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Visionary provides a portable, lightweight platform for in-browser rendering of diverse 3DGS variants and generative post-processing, serving reconstructive and generative world-modeling paradigms.

    Source: §6 · p. 13

    Evolving WebGPU and ONNX runtimes may differ in browser and OS compatibility, and browser CPU-memory constraints leave some post-processing offline.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

Neural rendering, particularly 3D Gaussian Splatting (3DGS), has evolved rapidly and become a key component for building world models. However, existing viewer solutions remain fragmented, heavy, or constrained by legacy pipelines, resulting in high deployment friction and limited support for dynamic content and generative models. In this work, we present Visionary, an open, web-native platform for real-time various Gaussian Splatting and meshes rendering. Built on an efficient WebGPU renderer with per-frame ONNX inference, Visionary enables dynamic neural processing while maintaining a lightweight, "click-to-run" browser experience. It introduces a standardized Gaussian Generator contract, which not only supports standard 3DGS rendering but also allows plug-and-play algorithms to generate or update Gaussians each frame. Such inference also enables us to apply feedforward generative post-processing. The platform further offers a plug in three.js library with a concise TypeScript API for seamless integration into existing web applications. Experiments show that, under identical 3DGS assets, Visionary achieves superior rendering efficiency compared to current Web viewers due to GPU-based primitive sorting. It already supports multiple variants, including MLP-based 3DGS, 4DGS, neural avatars, and style transformation or enhancement networks. By unifying inference and rendering directly in the browser, Visionary significantly lowers the barrier to reproduction, comparison, and deployment of 3DGS-family methods, serving as a unified World Model Carrier for both reconstructive and generative paradigms.

1 Introduction

Visionary addresses the deployment and extensibility gap in neural-rendering viewers with a browser-resident WebGPU runtime that couples per-frame ONNX inference and rendering. Its contract supports plug-in 3DGS-family algorithms, while the platform reports real-time performance and improvements over existing web renderers.

1 Introduction

  • The paper motivates a lightweight, flexible, unified platform for running heterogeneous 3DGS methods on consumer devices without low-level renderer rework.The target is to couple per-frame inference with high-throughput GPU rendering while maintaining browser real-time performance.Source: §1 · p. 2
  • Existing desktop and web viewers impose deployment friction or WebGL constraints that limit dynamic scenes and generative models, while demonstrations often rely on precomputed Gaussians or server-side inference.These constraints reduce interactivity and reproducibility.Source: §1 · p. 2
  • The Gaussian Generator contract specifies ONNX inputs, outputs, and metadata for generating or updating Gaussian attributes, allowing 3DGS-family methods to run as plug-ins without modifying the runtime.Specified attributes include position, scale, orientation, and color.Source: §1 · pp. 2–3
  • Visionary unifies browser-side compute and rendering across classic and MLP-based 3DGS, neural avatars, 4DGS, and generative post-processing networks.The pipeline supports dynamic workflows entirely on the client side.Source: §1 · pp. 2–3
  • Across representative static and dynamic scenes, Visionary sustains real-time interactive performance and improves frame time, dynamic update latency, and scalability over existing web-based renderers.The runtime also exposes a three.js plug-in and concise TypeScript API for integration into web applications.Source: §1 · p. 3

2 Related Works

Related work spans a rapidly diversifying 3DGS ecosystem, deployment-friction-prone viewers, and world models that increasingly use explicit 3D states. Visionary is positioned as an accessible runtime for viewing, comparing, and integrating these representations.

2.1 3D Gaussian Splatting

  • The 3DGS ecosystem now includes depth-aware, structured, large-scale, compressed, dynamic, avatar, and feed-forward reconstruction methods, motivating a unified viewer for comparison and mixing.3DGS offers fast rendering relative to NeRF, but city-scale environments expose training-efficiency limitations.Source: §2.1 · p. 3

2.2 3DGS Viewers and Plugins

  • Desktop viewers and engine plugins carry installation, driver, or version constraints, while browser viewers commonly rely on WebGL that limits large-scene scalability and client-side dynamic decoding or generative processing.Visionary is presented as combining web accessibility and WebGPU performance with support for advanced 3DGS variants and generative models through ONNX.Source: §2.2 · pp. 3–4

2.3 World Models and Interactive Generative Video

  • Video-based world models can exhibit 3D inconsistency, geometric hallucinations, and weak long-horizon stability; explicit 3D intermediate states are used to support physical plausibility and multi-view consistency.Visionary offers an ONNX-based pipeline for real-time inspection of explicit 3D states and is described as a testbed for physics-aware world models.Source: §2.3 · p. 4

3.1 Overview

Visionary is designed as a browser-native runtime in which WebGPU handles computation and rendering, ONNX contracts provide extensibility, and inference and rendering share a frame budget. Its pipeline combines decoded Gaussians with optional meshes before optional generative post-processing.

3.1 Overview

  • The design prioritizes browser execution through WebGPU, contract-driven algorithm integration without per-algorithm code branches, and inference-rendering coupling within one frame budget.The stated goals include zero-install sharing, on-device inference for privacy, and consistent behavior across operating systems.Source: §3.1 · p. 4
  • ONNX pre-decoding supports diverse 3DGS variants, after which generated Gaussians can be combined with optional meshes in a WebGPU hybrid renderer using depth-aware composition.The pipeline produces the final image before any optional generative post-processing.Source: Figure 2 · p. 5
  • ONNX-based generative models can optionally post-process the rendered output for tasks such as stylization and enhancement.The paper also describes generative-model integration through ONNX as a post-processing stage.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Definition of 3DGS

3DGS represents a scene as anisotropic 3D Gaussian primitives, projects them to 2D elliptical Gaussians, and composites their contributions by depth.

3.2 Definition of 3DGS

  • Each 3D Gaussian has a mean position, covariance derived from scale and rotation, opacity, and view-independent color encoded by spherical harmonics.The scene is represented as a set of these anisotropic primitives.Source: §3.2 · p. 5
  • A camera projection maps each 3D Gaussian to a 2D elliptical Gaussian.The projection uses its Jacobian at the Gaussian’s mean position.Source: §3.2 · p. 5
  • The rendered pixel color is computed by alpha-compositing Gaussian contributions after sorting them by camera depth.The formulation enables differentiable rendering and gradient-based optimization of Gaussian parameters.Source: §3.2 · p. 5

3.3 ONNX-based Gaussian Pre-decoding

Visionary separates method-specific Gaussian generation or deformation from a unified renderer through an ONNX pre-decoding contract. The section describes integrations for MLP-based 3DGS, 4DGS, and animatable avatars.

3.3 ONNX-based Gaussian Pre-decoding

  • The Gaussian Generator contract standardizes ONNX inputs, outputs, and metadata so plug-in algorithms can update Gaussian attributes each frame without method-specific renderer changes.The client supplies lightweight inputs and the ONNX graph outputs packed Gaussians for the WebGPU renderer; graph capture and chunked Concat/Split rewriting address runtime overhead and WebGPU limits.Source: §3.3 · pp. 5–6

3.3.1 Support for MLP-based 3DGS

  • Scaffold-GS anchors feed an MLP that generates Gaussian parameters for each view, and exporting the MLP and anchor data to ONNX enables on-the-fly browser inference for arbitrary camera poses.The generated neural Gaussians are rasterized using the standard 3DGS formulation; re-generating them every frame makes real-time online visualization challenging.Source: §3.3.1 · pp. 6–7

3.3.2 Support for 4DGS

  • The 4DGS integration exports a HexPlane deformation field and decoder to ONNX, taking timestamp t as its sole input and returning deformation residuals for canonical Gaussians.The six planes encode spatiotemporal features; a lightweight multi-head MLP predicts position offsets, rotation corrections, and scaling variations for real-time rendering.Source: §3.3.2 · p. 7

3.3.3 Support for Animatable Human Avatar

  • Animatable avatars use canonical Gaussians warped by body-pose and shape-driven skinning, with ONNX graphs outputting transformed Gaussian attributes for the rasterizer.The described implementations include SMPL-X pose and shape inputs, and R3Avatar frame-index input; the section also identifies LHM as an integrated example and says the formulation accommodates canonical-deformation methods.Source: §3.3.3 · pp. 7–8

3.4 WebGPU-based Hybrid Renderer

The hybrid renderer packs Gaussian attributes into GPU-friendly buffers and performs preprocessing, sorting, and rasterization in WebGPU. It can also compose Gaussian splats with mesh assets using depth-aware rendering.

3.4 WebGPU-based Hybrid Renderer

  • At the tested bicycle-scene scales, Figure 3 reports up to ∼100× speed-up over SparkJS and contrasts its dominant CPU sorting cost with Visionary’s low, stable GPU overhead.The comparison uses identical Gaussian complexity at full resolution and 1/2, 1/4, and 1/8 scales.Source: §3.4 · p. 8, Figure 3 · p. 9
  • Each frame, a compute kernel transforms Gaussians, culls invalid splats, computes their screen-space ellipses and depth keys, then appends valid splats to global buffers.Gaussian attributes are pre-packed, with covariance stored as an upper-triangular six-tuple and attributes packed to FP16 to reduce bandwidth.Source: §3.4 · pp. 8–9
  • Visionary radix-sorts splats by depth on the GPU and rasterizes them as instances, evaluating Gaussian weights and outputting premultiplied color.The vertex shader expands each splat into a screen-space quad.Source: §3.4 · p. 9
  • When a mesh is present, its depth buffer rejects occluded Gaussian fragments while visible splats are alpha-composited back-to-front.The mesh is rasterized first; Gaussian depth testing remains enabled while depth writing is disabled.Source: §3.4 · p. 9

3.5 ONNX-based Post-processing

The post-processing workflow exports a diffusion denoiser to a web-compatible ONNX graph, with timestep handling configured as fixed or dynamic. The exported graph is checked for valid inference shapes and model structure.

3.5 ONNX-based Post-processing

  • The workflow wraps the EXGS U-Net denoiser in an adapter that standardizes timestep handling, using either a fixed denoising-step buffer or a dynamic timestep input.The model is switched to evaluation mode and may be cast to FP16 on the target device.Source: §3.5 · p. 9
  • Shape-correct dummy inputs, optional dynamic axes, a sanity-check forward pass, and ONNX model validation complete the export workflow.The export uses torch.onnx.export() with specified input/output names, opset version, and constant folding, followed by onnx.checker.check_model().Source: §3.5 · p. 9

4 Experiments

Visionary combines WebGPU rendering with ONNX-based per-frame Gaussian generation, and is evaluated for efficiency, quality, compositing robustness, and dynamic-model inference. It reports faster rendering than SparkJS while preserving rendering quality and correct compositing under challenging viewing and multi-model conditions.

4.1 Setup

  • The evaluation compares representative WebGL viewers using identical 3DGS assets and camera trajectories, including static-scene efficiency and interactive compositing robustness.Experiments use an NVIDIA RTX 4090 GPU and Intel w5-3435X CPU.Source: §4.1 · pp. 9–10, §4.2 · p. 10

4.2 Runtime and Scalability vs. SparkJS

  • Up to ∼135× end-to-end speedup over SparkJS; at 6.062M Gaussians, sorting and total frame time fall from 172.87ms and 176.90ms to 0.58ms and 2.09ms.Visionary moves preprocessing and global sorting to WebGPU compute; Table 1 reports time analysis, while Figure 3 reports frame-time breakdown and end-to-end latency.Source: §4.2 · p. 10, Table 1 · p. 10

4.3 Rendering Quality

  • On MipNeRF360, Visionary matches or slightly improves SparkJS on PSNR, SSIM, and LPIPS.The paper attributes the slight improvement to avoiding SparkJS’s aggressive quantization and using compute shaders for 3DGS preprocessing.Source: §4.3 · p. 10, Table 2 · p. 10

4.4 Robustness under Rapid Viewpoint Changes

  • Visionary’s true global per-frame GPU sorting preserves correct back-to-front compositing during rapid camera motion, unlike SparkJS’s stale lazy ordering, which can produce visible artifacts.Figure 4 illustrates artifacts from SparkJS’s invalidated ordering during rapid viewpoint changes.Source: §4.4 · pp. 10–11, Figure 4 · p. 11

4.5 Composition Correctness vs. Supersplat

  • Visionary globally sorts valid splats across loaded models, avoiding the depth-inconsistent blending that local sorting can cause when partitions overlap.The unified global buffer supports compositing multiple 3DGS assets or 3DGS with a mesh in the same frame.Source: §4.5 · p. 11, Figure 5 · p. 11

4.6 Overhead of ONNX-based Gaussian pre-decoding

  • Per-frame decoding runs in real time for MLP-based 3DGS and 4DGS across scenes, while avatar inference takes 7–8ms depending on the model.Avatars typically require fewer Gaussians; Tables 3 and 4 cover inference times for MLP-based 3DGS/4DGS and avatars, respectively.Source: §4.6 · p. 12, Table 3 · p. 12, Table 4 · p. 12

5 Discussion

The discussion frames Visionary as an initial step toward world-model renderers that combine reconstructive and generative paradigms, and outlines possible extensions. It also notes browser-runtime compatibility and memory constraints that bound current in-browser execution.

5.1 Large World Model & Future

  • Visionary is presented as an initial step toward world-model renderers combining reconstructive, geometrically consistent representations with generative synthesis and open-ended dynamics.The discussion proposes a future closed loop between generation and reconstruction.Source: §5.1 · p. 12
  • Possible extensions include physical interaction, physics-aware modeling, spatially grounded agents, simulator interfaces, relighting, and domain adaptation for Sim-to-Real transfer.These are described as directions to explore, not current platform capabilities.Source: §5.1 · p. 12

5.2 Limitations

  • Evolving WebGPU and ONNX runtimes may cause browser and OS compatibility differences, while CPU memory constraints limit model size and leave some post-processing offline.Small- to medium-scale networks can currently be integrated into the pipeline.Source: §5.2 · p. 12

6 Conclusion

Visionary combines WebGPU with an ONNX-based integration contract to address neural-rendering platforms’ web-computation, extensibility, and dependency barriers. The conclusion presents it as a lightweight, research-oriented browser platform for diverse 3DGS variants and generative post-processing.

6 Conclusion

  • Visionary addresses limited web-native computation, extensibility, and heavy system dependencies in neural-rendering platforms.The conclusion describes its foundation as open and extensible.Source: §6 · p. 13
  • Combining WebGPU with a unified ONNX contract supports real-time in-browser rendering of diverse 3DGS variants and generative post-processing.Source: §6 · p. 13
  • The paper positions Visionary as a unified carrier for reconstructive and generative world modeling and a foundation for future spatial-intelligence applications.Source: §6 · p. 13

Related papers

Loading 2512.08478v1…