Source-linked AI summary
VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality
Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, Chenfanfu Jiang
TL;DR
Traditional 3D content creation is complex for non-experts, and interactive VR requires efficient rendering and deformation. VR-GS combines Gaussian Splatting, physics-based simulation, and two-level deformation embedding in a VR system. It reports improved immersion and realism, high usability, and positive user satisfaction, while retaining computational and manual-parameter limitations.
Problem
Traditional 3D content creation and interaction are time-intensive and expertise-dependent, motivating more accessible high-fidelity virtual content editing.
Method
VR-GS integrates Gaussian Splatting with physics-based simulation, two-level deformation embedding, and VR interaction for real-time physics-aware editing.
Results
Physics-based interaction scored 6.1 versus 4.8 for transform-based interaction on immersion and realism, while VR-GS achieved an SUS score of 83.5.
Takeaways & Limitations
VR-GS provides physically based dynamics, fine-grained manipulation, and an immersive interactive Gaussian Splatting experience in VR.
Takeaways & Limitations
High-fidelity rendering in large 2K scenes may introduce latency, and physical parameters are manually defined.
Abstract
from arXiv · showhide
As consumer Virtual Reality (VR) and Mixed Reality (MR) technologies gain momentum, there's a growing focus on the development of engagements with 3D virtual content. Unfortunately, traditional techniques for content creation, editing, and interaction within these virtual spaces are fraught with difficulties. They tend to be not only engineering-intensive but also require extensive expertise, which adds to the frustration and inefficiency in virtual object manipulation. Our proposed VR-GS system represents a leap forward in human-centered 3D content interaction, offering a seamless and intuitive user experience. By developing a physical dynamics-aware interactive Gaussian Splatting in a Virtual Reality setting, and constructing a highly efficient two-level embedding strategy alongside deformable body simulations, VR-GS ensures real-time execution with highly realistic dynamic responses. The components of our Virtual Reality system are designed for high efficiency and effectiveness, starting from detailed scene reconstruction and object segmentation, advancing through multi-view image in-painting, and extending to interactive physics-based editing. The system also incorporates real-time deformation embedding and dynamic shadow casting, ensuring a comprehensive and engaging virtual experience.Our project page is available at: https://yingjiang96.github.io/VR-GS/.
1 INTRODUCTION
Traditional 3D content creation is complex and time-intensive for non-experts, while NeRF can be inefficient for interactive applications. VR-GS addresses these challenges with physics-aware Gaussian Splatting interaction in VR.
- Traditional 3D/4D content creation relies on complex, time-intensive modeling tools and game engines that limit non-expert participation.
- NeRF’s volume rendering and inverse deformation mapping are inefficient for high-frame-rate interactive applications.
- VR-GS enables interactive, physics-aware manipulation of 3D content represented with Gaussian Splatting.
- Its two-level deformation embedding improves deformation realism and prevents undesirable spiky artifacts.
- VR-GS runs on VR devices with segmentation, inpainting, and shadow mapping for 3D content manipulation.
2 RELATED WORK
Related work spans explicit and implicit 3D representations, radiance-field editing, dynamic scene modeling, and real-time rendering. Existing methods still face limitations in unseen dynamics, latency, or geometric deformation.
- NeRF and other 3D representations support tasks including synthesis, manipulation, animation, reconstruction, and data transmission.
- Radiance-field editing methods provide shape, style, sketch, and texture manipulation, but target specific editing capabilities.
- Time-conditioned dynamic NeRFs and 4D Gaussian primitives generally model motions captured from input data rather than unseen dynamics.
- Physics-based simulations integrated with static 3D representations enable physically plausible and novel dynamic scenes.
- NeRF rendering remains computationally expensive for real-time applications, motivating gaze-contingent, shading, and hybrid representation techniques.
- Interactive NeRF systems support manipulation and visibility edits but do not deform virtual objects’ geometrical structures.
3 SYSTEM DESIGN
VR-GS combines Gaussian Splatting, physics simulation, and VR interaction in a unified real-time framework. Its design targets realistic physics-based editing while preserving low latency and natural visual feedback.
- VR-GS proposes a unified physics dynamics-aware VR system for real-time interactions with Gaussian Splatting.
- The system uses physics-based simulation to produce realistic dynamics and facilitate 3D shape editing in VR.
- Low latency is treated as crucial for preventing disorientation or motion sickness in interactive VR systems.
- VR-GS uses a reduced representation and parallel-friendly algorithms, replacing per-Gaussian discretization with a tetrahedral approach for real-time performance.
- A two-level interpolation scheme embeds Gaussians within bounding tetrahedra and then within a simulated cage mesh to smooth deformation fields and reduce spiky artifacts.
- The framework adds interaction mechanisms intended to provide natural object movement, realistic appearances, and real-time feedback.
4.1 Gaussian Splatting
Gaussian Splatting represents scenes explicitly with anisotropic Gaussian kernels and renders views through depth-ordered alpha blending. This explicit, efficient representation supports post-processing and VR integration.
- Gaussian Splatting encodes 3D scene information with anisotropic Gaussian kernels containing learnable means, opacities, covariances, and spherical harmonic coefficients.
- The renderer projects 3D splats into 2D screen space, orders them by z-depth, and computes pixel color through near-to-far alpha blending.
- Each kernel’s contribution combines view-dependent spherical-harmonic color with opacity and its evaluated 2D Gaussian weight.
- The VR-GS pipeline uses Gaussian kernels with VDB, bounding mesh reconstruction, and tetrahedralization before simulation and rendering.
- Differentiable rendering allows image loss to backpropagate to Gaussian parameters for optimization.
- Compared with implicit NeRF representations, Gaussian Splatting supports post-processing manipulation and offers efficient rasterization with high rendering quality for VR.
4.2 VR-GS Assets Preparation
VR-GS prepares high-fidelity Gaussian Splatting assets for interactive editing through segmentation, inpainting, and simulation-mesh construction. These offline preparations support object-level manipulation while preserving real-time performance.
- Asset preparation: Each VR asset combines a high-fidelity 3D Gaussian Splatting reconstruction with a moderate-resolution simulatable tetrahedral mesh.The mesh is prepared offline to enable real-time physics-aware dynamics without being rendered during interaction.
- Segmentation: Segmented Gaussian Splatting reconstructions allow users to interact with individual objects in a large scene.The system generates consistent colored 2D masks across views and adds learnable RGB attributes to Gaussian kernels.
- Segmentation: The segmentation loss compares colored 2D segmentation results with rendered Gaussian attributes, while the total loss combines reconstruction and segmentation objectives.The reconstruction terms use normally rendered images and multi-view ground truths; the experiments set λ=0.2 and λseg=0.1.
- Inpainting: 2D LaMa inpainting guides 3D Gaussian inpainting after object removal exposes previously occluded holes.Gaussian kernels outside holes are frozen while a kernel patch is optimized using an inpainting loss.
- Mesh generation: A tetrahedral simulation mesh is generated for each segmented object, and two-level embedding places each Gaussian kernel inside a local tetrahedron.The local tetrahedra are independently embedded into the global mesh.
4.3 Unified Framework for Simulation and Rendering
VR-GS unifies Gaussian rendering with mesh-based physics simulation by deforming Gaussian kernels through an embedded tetrahedral representation. Its two-level embedding improves deformation smoothness, while dynamic shadow mapping preserves spatial cues during interaction.
- Simulation: XPBD replaces the per-Gaussian simulator to provide real-time interactions using strain-energy elasticity and velocity-based damping.Physical parameters such as Young’s modulus, Poisson ratio, and density are manually tuned for visually plausible dynamics.
- Deformation embedding: The piecewise-linear mesh deformation produces a piecewise-constant deformation gradient within each tetrahedron.The inverse rest-shape basis can be computed before simulation.
- Deformation embedding: Gaussian means and covariances are deformed from barycentric coordinates and the initial covariance matrix, then rendered directly by point splatting.Directly embedding Gaussian centers may leave ellipses insufficiently enclosed by simulation tetrahedra, producing spiky artifacts.
- Two-level embedding: Two-level embedding independently envelopes each Gaussian with a tight local tetrahedron and embeds that tetrahedron’s vertices into the global simulation mesh.Averaging deformation across surrounding global tetrahedra produces a smoother deformation field and reduces sharp artifacts.
- Dynamic shadows: Dynamic shadow mapping estimates depth from the light source and tests Gaussian visibility to keep shadows aligned with moving or deforming objects.The shadow provides spatial-distance cues during manipulation and is compatible with the Gaussian Splatting rasterization pipeline.
5 EVALUATION
The evaluation examines two-level embedding, inpainting, and shadow mapping as key components of VR-GS. The reported comparisons show reduced deformation artifacts, more complete reconstructed scenes, and added time-dependent shadows.
- Two-level Embedding: The two-level embedding ablation evaluates whether the method reduces spiky artifacts in deformed Gaussian kernels.The comparison contrasts naive closest-tetrahedron embedding with localized tetrahedra whose vertices are embedded into the cage mesh.
- Two-level Embedding: Under extreme stretching or twisting, naive embedding produces severe spiky artifacts, whereas two-level embedding yields smoother deformation gradients.The Gaussian deformation is determined by averaging deformations at the local tetrahedron’s vertices.
- Inpainting: Gaussian Splatting leaves unseen object and background regions unreconstructed, causing black-hole artifacts when foreground objects move.This limitation arises because reconstruction covers surfaces visible in the multi-view training images.
- Inpainting: LAMA-generated inpainted multi-view images guide fine-tuning and 3D Gaussian inpainting to produce more complete and realistic scenes.The method combines object segmentation masks with 2D inpainting before updating the 3D representation.
- Shadow Map: Original Gaussian Splatting represents shadows as textures and therefore lacks dynamic shadows when objects move, while VR-GS adds time-dependent shadow maps.Users can select light-source position and direction to reproduce the original scene’s lighting setting.
6 EXPERIMENT
VR-GS is evaluated as a real-time physics-aware Gaussian Splatting system through performance comparisons, visual-quality assessment, and interactive VR demonstrations. The experiments examine simulation trade-offs and report efficient preparation of interactive scenes.
- Experimental Setup: VR-GS uses a Unity prototype with a CUDA plugin, tested on a Quest Pro HMD and an RTX 4090-equipped desktop.The hardware includes an Intel Core i9-14900KF CPU and 32GB memory.
- Performance Trade-offs: Coarse cage meshes increase frame rates but can lose fine details, while finer meshes increase computational cost and require more XPBD iterations.The system constrains mesh resolution to 10K–30K vertices to balance frame rate and physical accuracy.
- Additional Components: Dynamic shadow casting replaces static Gaussian Splatting shadow textures to support more immersive interactive scenes.The system also includes segmentation, inpainting, and shadow-map processing for VR content manipulation.
- Method Comparison: VR-GS matches PhysGaussian in visual quality and outperforms PAC-NeRF in clarity and realism while enabling significantly faster simulations.The comparison uses standardized frame and simulation step times, with VR-GS using one XPBD iteration per substep.
- VR Demonstrations: The system supports physics-based manipulation of reconstructed foxes, bears, horses, plush toys, elastic rings, rigid bricks, and reconstructed human animations.Demonstrations include object insertion into existing scenes and motion generation from rigged human surface meshes.
7 USER STUDY
A user study with 10 participants compared physics-based and transform-based interaction and assessed open-ended editing in VR. Participants preferred physics-based interaction for immersion and realism and rated the system highly for usability and satisfaction.
- Study Design: The study included 10 participants, comprising 2 3D VFX professionals and 8 novices, who completed goal-directed and open-ended VR tasks.The study used a tutorial, Latin-square session ordering, and questionnaires covering usability and subjective experience.
- Study Materials: The study included reconstructed human animation as an interactive demonstration alongside object manipulation and game scenarios.The Just Dance demonstration generated motion from a reconstructed human body.
- Results and Discussion: Physics-based interaction significantly improved immersion and realism over transform-based interaction, scoring 6.1 versus 4.8 on average (p = .0227).Participants played ring-toss and toy-collection games under both interaction configurations.
- Results and Discussion: Users rated VR-GS 4.6/5 for ease of use and 4.8/5 for overall satisfaction, with an SUS score of 83.5 classified as excellent.Participants also reported positive reactions to virtual-content placement, lighting, and generated dynamics.
8 CONCLUSION AND FUTURE WORK
VR-GS presents a physical dynamics-aware interactive Gaussian Splatting system for editing high-fidelity virtual content in real time. Its future development is constrained by VR rendering demands and manually defined physical parameters, with broader material support proposed.
- Conclusion: VR-GS combines physically based dynamics with fine-grained interaction and manipulation control for real-time high-fidelity virtual-content editing.The conclusion describes the system as addressing challenges in editing realistic virtual content.
- Limitations and Future Work: Rendering high-fidelity Gaussian kernels in large 2K-resolution VR scenes may cause latency because of computational demands.The authors identify this as a current limitation despite positive participant feedback.
- Limitations and Future Work: Physical parameters are manually defined, motivating future automation through video-based estimation or large-vision models.Future work also considers fluid and cloth materials and multimodal assessment of generated-dynamics fidelity.