Source-linked AI summary
Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting
Ziyuan Xia, Jingyi Xu, Chong Cui, Yuanhong Yu, Jiazhao Zhang, Qingsong Yan, Tao Ni, Junbo Chen, Xiaowei Zhou, Hujun Bao, Ruizhen Hu, Sida Peng
TL;DR
Embodied AI simulators need realistic visual observations and dynamic humans, but mesh-based systems provide limited fidelity and avatar support. Habitat-GS integrates 3DGS scene rendering and drivable gaussian avatars into Habitat-Sim while preserving ecosystem compatibility. Experiments report stronger cross-domain generalization, effective human-aware navigation, and scalable performance, with mixed-domain training most effective.
Problem
Mesh-based simulators offer limited visual realism and constrained dynamic-human support, restricting simulation for realistic human-populated navigation.
Method
Habitat-GS extends Habitat-Sim with real-time photorealistic 3DGS rendering, scalable asset import, and gaussian avatars that act as visual entities and navigation obstacles.
Results
Experiments report stronger cross-domain generalization with mixed-domain training, effective avatar-aware navigation, and scalability across scene complexity and avatar counts.
Takeaways & Limitations
Habitat-GS provides an open-source, Habitat-compatible foundation for high-fidelity embodied-AI training with dynamic human-aware navigation.
Takeaways & Limitations
Pure GS training requires considerably more steps to converge and is not cost-effective under practical compute constraints within the fixed budget.
Abstract
from arXiv · showhide
Training embodied AI agents depends critically on the visual fidelity of simulation environments and the ability to model dynamic humans. Current simulators rely on mesh-based rasterization with limited visual realism, and their support for dynamic human avatars, where available, is constrained to mesh representations, hindering agent generalization to human-populated real-world scenarios. We present Habitat-GS, a navigation-centric embodied AI simulator extended from Habitat-Sim that integrates 3D Gaussian Splatting scene rendering and drivable gaussian avatars while maintaining full compatibility with the Habitat ecosystem. Our system implements a 3DGS renderer for real-time photorealistic rendering and supports scalable 3DGS asset import from diverse sources. For dynamic human modeling, we introduce a gaussian avatar module that enables each avatar to simultaneously serve as a photorealistic visual entity and an effective navigation obstacle, allowing agents to learn human-aware behaviors in realistic settings. Experiments on point-goal navigation demonstrate that agents trained on 3DGS scenes achieve stronger cross-domain generalization, with mixed-domain training being the most effective strategy. Evaluations on avatar-aware navigation further confirm that gaussian avatars enable effective human-aware navigation. Finally, performance benchmarks validate the system's scalability across varying scene complexity and avatar counts.
1 Introduction
Habitat-GS addresses limited simulator visual fidelity and dynamic-human support by integrating 3DGS rendering and drivable gaussian avatars into Habitat-Sim. Experiments report stronger cross-domain generalization, effective avatar-aware navigation, and scalable performance.
- Motivation: Existing simulators use mesh rasterization, limiting high-frequency and view-dependent visual realism and constraining dynamic human avatar support.These limitations contribute to the Sim-to-Real gap and hinder navigation research in human-populated environments.
- Approach: Habitat-GS upgrades Habitat-Sim with real-time photorealistic 3DGS rendering, scalable asset import, and full Habitat ecosystem compatibility.The platform is open-source and retains Habitat-Lab task, training, and evaluation APIs.
- Results: Experiments show 3DGS training improves cross-domain generalization, with mixed-domain training producing the most effective strategy.The evaluation includes VLM-based scene assessment and point-goal navigation experiments.
- Results: Avatar-aware experiments show gaussian avatars enhance navigation in human-populated scenarios, while benchmarks confirm scalability across scene complexity and avatar counts.The reported validation spans multiple navigation paradigms and system-performance settings.
- Approach: Its gaussian avatar module combines photorealistic rendering with collision properties for dynamic-human navigation.Avatars support natural motion and navigation-level obstacle avoidance.
2 Related Work
Related work spans mesh-based embodied simulators, neural rendering, 3DGS, and gaussian avatars. Habitat-GS combines 3DGS rendering and drivable gaussian avatars in an open-source platform deployable on standard datacenter accelerators.
- Embodied AI Simulators: Embodied AI simulators predominantly use mesh-based rasterization, while humanoid representations generally lack the visual fidelity needed for fine-grained human appearance and motion cues.The comparison considers rendering assets, avatar support, openness, and hardware requirements.
- Habitat-GS: Habitat-GS combines photorealistic 3DGS rendering, dynamic high-fidelity gaussian avatars, and a mature open-source Habitat ecosystem.The system preserves traditional NavMesh navigation while upgrading scene rendering and avatar support.
- Neural Rendering: NeRF synthesizes photorealistic novel views but requires per-pixel ray marching and is difficult to integrate with existing simulator rendering backends.These constraints limit its suitability for real-time embodied-AI sensor pipelines.
- Habitat-GS: Table 1 characterizes GPU requirements beyond standard CUDA-capable GPUs and identifies Habitat-GS as combining 3DGS, drivable gaussian avatars, openness, and standard datacenter deployment.The table’s “No Special Req.” category denotes that standard GPUs are sufficient.
- 3D Gaussian Splatting: 3DGS uses explicit anisotropic 3D Gaussians and tile-based rasterization to support real-time, high-fidelity rendering and integration with traditional graphics pipelines.Its explicit representation also facilitates spatial editing, asset composition, and dynamic deformation.
- System Architecture: The system overview separates asset preparation, simulation rendering, NavMesh obstacle blocking, and the embodied agent loop producing RGB-D observations and rewards.Scene gaussians are rasterized directly, while avatar gaussians undergo LBS deformation before rasterization.
3 Habitat-GS Simulation Environment
Habitat-GS combines 3DGS rendering with drivable gaussian avatars while preserving Habitat’s navigation infrastructure. Its design separates visual rendering from navigation geometry and supports dynamic avatar interaction through synchronized rendering and collision mechanisms.
- Architecture: Habitat-GS extends Habitat-Sim with 3DGS scene rendering and a gaussian avatar module, while retaining traditional NavMeshes for navigation.This visual–navigation decoupling addresses the lack of explicit surface geometry in 3DGS.
- Asset Preparation: The pipeline prepares Gaussian assets, avatar attributes, motion trajectories, and proxy capsules before simulation.Proxy capsules approximate per-frame avatar collision geometry, while trajectories provide pose transformations.
- Simulation Environment: At each frame, CUDA renderers rasterize scene and posed avatar Gaussians, while CUDA–OpenGL interoperation transfers RGB-D outputs without CPU-side movement.Scene and avatar rendering run in parallel, with depth compositing supporting heterogeneous assets.
- Embodied Agent Loop: The embodied-agent loop feeds composited RGB-D observations to the policy, while augmented navigation clips actions at avatar collision boundaries.Avatar query APIs expose clearance distances and collision flags for rewards and metrics.
- Dynamic Gaussian Avatar Module: GAMMA generates scene-aware walking trajectories, and dynamic capsule obstacles update the path planner to prevent agents from passing through avatars.The avatar manager injects current capsule positions into planning and checks agent–avatar capsule intersections each step.
- Dynamic Gaussian Avatar Module: Compared with mesh avatars, gaussian avatars preserve finer visual details, including clothing wrinkles and hair texture.The module combines photorealistic avatar appearance with navigation-obstacle behavior.
4 Experiments
Experiments evaluate scene quality, navigation generalization, avatar-aware behavior, and runtime scalability. 3DGS scenes improve visual quality and support robust cross-domain navigation, while mixed-domain training and gaussian-avatar fine-tuning provide the strongest practical outcomes.
- Experimental Setup: The experiments assess 3DGS scene quality, PointNav generalization, avatar-aware navigation, and efficiency under varying scene and avatar counts.The evaluation uses disjoint mesh and GS test collections, standard navigation metrics, and avatar-specific collision and personal-space metrics.
- 3DGS Scene Quality: GS scenes substantially outperform mesh scenes across rendering quality, realism, and scene diversity in the VLM assessment.Gemini 3.0 Pro scores 240 screenshots per renderer on three 10-point perceptual dimensions.
- PointNav Generalization: Mesh-only training converges quickly but plateaus near 0.6 success rate and generalizes poorly to GS test scenes.The reported limitation is attributed to insufficient visual realism in mesh environments.
- PointNav Generalization: GS-only training reaches ∼0.8 SR on its training set and exceeds the mesh-only baseline on GS tests, but requires more steps to converge.Within the fixed 5×10^7-step budget, it remains below Config A on mesh tests and is not cost-effective under practical compute constraints.
- PointNav Generalization: Mixed-domain training is most effective: Config E, using 20 mesh and 80 GS scenes, achieves the best GS test performance while matching the mesh-only baseline on mesh tests.Mesh scenes provide geometric learning, whereas GS scenes add visual robustness that transfers across domains.
- Avatar-Aware Navigation: Avatar training lowers collision and personal-space intrusion rates on both GS and mesh dynamic test sets after only 5×10^6 fine-tuning steps.On mesh tests, CR falls from 2.521% to 2.342% and PSI from 0.075 to 0.068.
5 Conclusion
Habitat-GS extends Habitat-Sim with 3D Gaussian Splatting and photorealistic drivable gaussian avatars while preserving Habitat compatibility. Its visual–navigation decoupling supports navigation but limits physical interaction to obstacle avoidance.
- Habitat-GS upgrades Habitat-Sim’s visual backbone from mesh rasterization to 3D Gaussian Splatting and adds photorealistic drivable gaussian avatars.
- The system is fully open-sourced and remains compatible with Habitat-Lab tasks, training, and evaluation APIs.
- Visual–navigation decoupling confines physical interaction to navigation-level obstacle avoidance rather than force- or impulse-level contact.
- Tasks requiring fine-grained physical interaction, such as grasping or pushing Gaussian-Splatting objects, remain outside Habitat-GS’s current scope.
A Habitat-GS Architecture Details
Habitat-GS integrates 3DGS rendering and dynamic gaussian avatars into Habitat-Sim through modular import, rendering, deformation, and sensor-pipeline components. Its architecture separates photorealistic visual processing from NavMesh-based navigation while maintaining real-time data flow.
- System architecture: The architecture separates CUDA-based 3DGS rendering and avatar deformation from NavMesh navigation with injected proxy capsules.
- Asset integration: Custom importers parse binary .ply Gaussian-Splatting scenes and unified .npz avatar representations from diverse asset sources.
- Rendering integration: GaussianSplattingDrawable and GaussianAvatarDrawable integrate static scenes and dynamic avatars into Habitat-Sim’s shared drawable interface.
- Rendering integration: Zero-copy CUDA–OpenGL interoperability writes RGB and depth outputs directly into mapped buffers, avoiding CPU–GPU transfers during rendering.
- Avatar deformation: Precomputed joint transformations and CUDA Linear Blend Skinning deform Gaussian avatars at runtime without costly neural-network inference.
A.4 Dynamic Navigation Module
The dynamic navigation module converts Gaussian avatars’ motion into lightweight capsule obstacles because 3DGS lacks explicit surfaces for direct collision detection. Precomputed capsules are interpolated and injected into the NavMesh alongside photorealistic avatar rendering.
- Dynamic navigation module: Because 3DGS lacks explicit geometric surfaces, Habitat-GS uses proxy capsules to enable navigation-level obstacle avoidance.
- Offline pre-computation: Offline trajectory synthesis generates one capsule per skeletal bone segment and stores capsule sequences across frames.
- Runtime injection: At runtime, current capsule positions are obtained by temporal interpolation and merged across active avatars for NavMesh processing.
- Design trade-off: The Gaussian representation provides photorealistic rendering while precomputed capsule proxies provide efficient navigation collision primitives with low runtime overhead.
- Qualitative results: Qualitative visualizations show diverse 3DGS scenes with high-fidelity dynamic avatars and navigation episodes involving goal-directed planning and human-aware obstacle avoidance.
C.1 Additional Results on Static Navigation
Fully converged PointNav evaluations compare mesh-only, GS-only, and mixed-domain training across mesh and GS test sets. Mesh-only training is narrowly in-domain, GS-only training transfers better, and mixed-domain configurations achieve the strongest overall generalization.
- Experimental setup: 1×10^8 steps fully converge all five PointNav training configurations, enabling comparison of final capability rather than training efficiency.
- Experimental setup: The Average column measures overall cross-domain generalization by averaging each metric across mesh and GS test sets.
- Training-domain comparison: Mesh-only Config A achieves the best mesh-test SPL and lowest mesh-test DTG but the lowest GS-test SR (61.70) and SPL (52.06).
- Training-domain comparison: GS-only Config B reaches GS-test SR 77.60 and SPL 66.96 versus Config A’s 61.70 and 52.06, while mesh-test SR remains 59.80 versus 62.80.
- Mixed-domain training: Mixed-domain Configs D and E dominate the Average columns, tying for best average SR while D leads average SPL and E leads average DTG.
C.2 Avatar-aware PointNav Task Definition
Avatar-aware PointNav retains the standard point-goal task while adding dynamic-avatar safety constraints. Signed clearance, intrusion, collision, and reward terms encourage efficient navigation that respects avatars’ personal space.
- Task interpretation: Avatar-aware PointNav preserves the standard goal and action interface while treating avatars as moving obstacles with dense safety penalties.The task remains point-goal navigation, but successful behavior must reach the goal while respecting nearby avatars’ personal space.
- Signed clearance to avatars: Signed clearance measures separation from the nearest avatar capsule, with positive, zero, and negative values indicating separation, contact, and overlap.When no avatar is active, clearance is +∞ and the avatar-related penalty is zero; collisions use ε_col = 10^-5.
- Personal intrusion function: Personal intrusion is zero outside the personal-space radius, increases linearly within it, and saturates at d_int during overlap.The experiments set d_int = 1.0 m and d_crit = 0.5 m, keeping intrusion bounded and numerically stable.
- Two-stage avatar penalty: The avatar penalty uses a smooth two-stage function that weakly discourages boundary grazing, sharply penalizes critical-zone proximity, and saturates at collision.The branches are continuous at c_t = d_crit and the inner branch saturates smoothly for c_t ≤ 0.
- Total reward: Total reward linearly combines standard PointNav progress with avatar safety, then adds Habitat slack and success terms.The experiments use λ_prog = 1.0 and λ_avatar = 1.0; the agent is rewarded for goal progress and penalized for dangerous proximity or collision.
C.3 Human Tracking Experiment
TrackNav evaluates whether agents can follow moving avatars from an appropriate distance and orientation. The experiment compares mesh-trained and Gaussian-trained agents across mesh and GS test domains, emphasizing collision safety and transferability.
- Task definition: TrackNav requires continuously following a moving avatar within a prescribed distance band while maintaining visual contact from behind.The task has no fixed goal position and tests longer-horizon human-aware navigation.
- Tracking definition: A successful tracking step requires simultaneous distance-band, view-cone, and rear-sector conditions.The experiment uses a 1.2–2.5 m distance band, a 45° view cone, and a 60° rear-sector constraint.
- Tracking definition: Track rate is the fraction of steps satisfying all three tracking conditions over the episode.The binary tracking indicator is aggregated as TR = Σ_t track_t / T.
- Reward design: The TrackNav reward combines approach-to-band progress, an in-band Gaussian-shaped bonus, collision penalties, orientation penalties, anti-circling shaping, and tracking bonuses.These terms reward reaching and maintaining the desired relationship while discouraging unsafe proximity and orbiting.
- Analysis: GS-trained agents provide a safer and more generalizable balance, including higher TR on the GS test set than their mesh-trained counterpart.The study trains separate Mesh Scene + Mesh Avatar and GS Scene + GS Avatar agents and cross-evaluates both configurations.
- Analysis: Mesh-trained agents attain higher tracking rates but unsafe collisions, with average CC 11.93 versus 5.68 for GS-trained agents.The mesh policy’s TR drops from 38.40 on mesh tests to 17.78 on GS tests, whereas GS-trained behavior remains more stable across domains.
C.4 Text Prompt for VLM Evaluation
The VLM evaluation prompt scores rendered simulator images for rendering quality, realism, and scene diversity. It standardizes assessment across exactly ten randomly sampled images using 1–10 ratings and strict JSON output.
- Evaluation setup: The assessment presents exactly 10 randomly sampled mesh and GS rendered scene images to Gemini 3.0 Pro.The prompt is designed to elicit objective scene-quality scores from the sampled batch.
- Evaluation criteria: Each image receives 1–10 scores for rendering quality, realism, and scene diversity, with higher scores indicating better results.Rendering quality covers fidelity and artifacts, realism compares scenes with real-world appearance, and diversity compares images within the batch.
- Evaluation protocol: The prompt instructs the evaluator not to infer rendering methods, engines, or dataset names.This constrains scoring to visible image properties rather than presumed implementation details.
- Output format: The evaluator must return strict JSON containing per-image positions, three integer scores, and a brief reason of at most 60 words.The required output enumerates image positions 1 through 10.