PaperlayerView exact revision on arXiv

Source-linked AI summary

CognitiveReality: Robot-Agnostic Semantic Gaussian Mapping with an LLM Agent for Immersive Collaborative VR Teleoperation

Timofei Kozlov, Dmitrii Maliukov, Andrey Marchenko, Dmitrii Plotnikov, Miguel Altamirano Cabrera, Dzmitry Tsetserukou

arXiv:2609.31418v1cs.RO

TL;DR

Photorealistic 3D teleoperation views remain passive, offering no intuitive way to understand scene contents, refer to them, or interact with a robot. CognitiveReality combines live Gaussian-SDF reconstruction with a semantic Scene Model and a tool-using agent for VR interaction. Evaluations report improved tool exact match and successful live navigation and re-observation trials.

  • Problem

    Photorealistic reconstructions remain passive representations without an intuitive way for users to understand scene contents, refer to them, or interact with the robot.

    Source: §I · p. 1

  • Method

    CognitiveReality combines real-time Gaussian-SDF reconstruction with a VLM agent that uses persistent semantic object state and typed tools for scene interaction and robot control.

    Source: §I · pp. 1–2

  • Results

    The deployed router reached 81.24% tool exact match, while live trials reached 26 of 30 marked navigation goals and completed all 20 re-observation requests.

    Source: §V · p. 8

  • Takeaways & Limitations

    Persistent object references connect language, pointing, visualization, and robot-directed requests while allowing failure modes to be evaluated separately.

    Source: §V · p. 8

Abstract

from arXiv · show

A photorealistic 3D view tells a teleoperator where a robot is, but not what the scene contains, how well each object has been observed, or how to turn pointing and speech into robot action. CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent. One mapper binary serves any platform through configuration alone: it ingests poses from robot SLAM, joint kinematics, motion capture or an inline visual tracker, bridges localization outages with a shadow tracker and keyframe-anchored PnP, and maintains open-vocabulary instance identities with per-object quality at 2 Hz. Speech and controller rays are grounded against persistent scene objects through validated typed tools and operator-confirmed robot actions. In the controlled agent evaluation, the deployed local Qwen3-VL-8B router reaches 81.24\% tool exact match, while merge-aware replay correctly redirects 101 absorbed object identifiers. On robot data CognitiveReality exceeds a Gaussian-plus-SDF baseline by 2-8 dB; pose error through 5-40 s SLAM outages stays within 1-8 cm. Deployed live on two quadrupeds, the agent executed 26 of 30 navigation requests and 20 of 20 re-observation requests, raising object quality by 2-5 dB.

I. INTRODUCTION

Photorealistic 3D views remain passive, leaving users without an intuitive way to understand scene contents, refer to them, or interact with the robot. CognitiveReality makes an evolving object-level map the shared interface for an agent and VR operator, combining reconstruction, semantic state, typed actions, and immersive control.

I. INTRODUCTION

  • CognitiveReality combines real-time Gaussian-SDF reconstruction, a VLM-powered agent, and immersive VR interaction for scene understanding, object interaction, and robot control.The agent mediates natural-language and high-level commands in the reconstructed scene.Source: §I · p. 1
  • The mapper switches at runtime between robot SLAM poses and an inline visual tracker, with blur filtering and noise-weighted depth fusion; it exceeds the Gaussian-plus-SDF baseline by 2-8 dB on robot data.Supervision frames may be gyroscope-filtered, and depth fusion uses a sensor noise model.Source: §I · p. 2
  • A shadow tracker bridges robot-localization outages, while keyframe-anchored PnP bridges simultaneous visual-tracker loss; pose error stays within 1-8 cm through 5-40 s outages.The map rendered during the outage window is described as indistinguishable from an uninterrupted reference.Source: §I · p. 2
  • Open-vocabulary instance identities persist on map Gaussians, and semantics run at 2 Hz without dropping mapper frames; the Scene Model exposes object class, position, coverage, and rendering quality.The agent and operator use this object list during the mission.Source: §I · p. 2
  • Typed tools ground speech and pointing in the live Scene Model, with pre-routing, scene validation, and operator confirmation checking structured calls before execution.The interface was evaluated through command cases, temporal reference replay, metadata controls, and live robot execution.Source: §I · p. 2
  • The VR interface supports controller-ray instance and floor-goal selection, ray-grounded push-to-talk, and route, trail, highlight, and quality overlays in a 90 Hz host-rendered stereo stream.A 3D robot model is animated or mirrors physical joint movements; the headset-to-robot loop is demonstrated across platforms with and without onboard SLAM.Source: §I · p. 2

II. RELATED WORK

Prior work spans online geometric and semantic mapping, immersive Gaussian-scene teleoperation, and headset rendering. CognitiveReality combines these threads with persistent object identities and quality attributes shared across robot, agent, and operator.

A. Online Reconstruction and Semantic Mapping

  • Hybrid Gaussian-SDF methods build on photorealistic 3D Gaussian Splatting; CognitiveReality adds interchangeable robot localization, inline ORB-SLAM3, vibration-gated supervision, uncertainty-weighted depth fusion, and outage recovery.The passage identifies GPS-SLAM as an example of hybrid Gaussian-SDF methods.Source: A. Online Reconstruction and Semantic… · p. 2
  • Open-vocabulary mapping includes object graphs such as ConceptGraphs, language-feature Gaussians in LangSplat, and instance identities in Gaussian Grouping.Online examples include OpenGS-SLAM, Online Language Splatting, and OnlinePG.Source: A. Online Reconstruction and Semantic… · p. 2

B. Gaussian Scenes and Immersive Teleoperation

  • Immersive Gaussian-scene systems include RFFR, Boehringer et al., HIL-GS, VRSplat, and GaussAnything [14]; CognitiveReality supports remote and headset-local rendering with common object IDs.Those IDs connect visualization, language grounding, highlighting, and robot actions.Source: B. Gaussian Scenes and Immersive… · p. 2

C. Tool Use and Multimodal Human-Robot Interaction

The paper situates typed language-and-pointing tools within agent-world interaction, then describes a live Gaussian-SDF map that maintains semantic identities and recovers tracking across outages. Semantic-only Label Splats and an asynchronous auditor extend coverage and object discovery.

C. Tool Use and Multimodal Human-Robot Interaction

  • Typed action interfaces distinguish action selection from constructing valid, argument-complete calls,,; CognitiveReality combines these with speech-and-pointing interaction and evaluates call correctness, execution, grounding, and reference validity.Source: C. Tool Use and Multimodal Human-Robot… · p. 2

D. Agent-World Interaction

  • Prior agent-world systems use scene graphs, object-centric typed tools [20], Gaussian-map memory [21], frontier selection [22], or semantic exploration; CognitiveReality anchors actions in a live robot-built map viewed from the operator’s perspective.Its object model includes coverage and rendering quality, and actions pass through validated typed tools.Source: D. Agent-World Interaction · p. 2

A. Online Tracking and Reconstruction

  • The mapper fuses depth into an SDF and optimizes 3D Gaussians over its surface for photorealistic rendering; consistency filters and asynchronous optimization support real-time frame acquisition.Candidate pixels are filtered using raycast depth and TSDF confidence, and poorly constrained primitives are pruned.Source: A. Online Tracking and Reconstruction · pp. 2–3
  • During primary tracking outages, a secondary visual tracker estimates alignment to the global map; if both pose sources fail, PnP-RANSAC registers frames against nearby RGB-D keyframes.Stored keyframes anchor the tracking chain, so the global map is not reset when the external pose source recovers.Source: A. Online Tracking and Reconstruction · p. 3
  • Label Splats place semantic-only Gaussians on the SDF surface after two-view geometric consistency checks, while excluding them from photometric rendering to preserve fidelity.They provide 100% semantic coverage on Replica office0 versus 47.6% without Label Splats, and improve object-centering votes for approach and viewpoint planning.Source: A. Online Tracking and Reconstruction · p. 3
  • An asynchronous semantic auditor tracks at 2-3 Hz and discovers objects at 0.5 Hz, using a local vision-language model to verify masks and identify novel objects.Novel objects are registered within 1-2 s without strict viewpoint confirmation, or 4-5 s with a two-view confirmation constraint.Source: A. Online Tracking and Reconstruction · pp. 3–4

B. VR Rendering and Streaming

CognitiveReality provides remote stereo image streaming and headset-local scene rendering. The modes trade headset resource use and network-sensitive head-motion response against local compute, memory, and cache-dependent visual quality.

B. VR Rendering and Streaming

  • Remote stereo image streaming has the host render both eye views from headset pose and projection matrices, combining TSDF raycasting, Gaussian appearance, and object highlighting before WebRTC delivery.This mode reduces headset rendering and memory load, but head-motion response depends on network delay, host rendering, image encoding, and frame delivery.Source: B. VR Rendering and Streaming · p. 4
  • Remote streaming reduces headset resource demands, whereas local rendering avoids per-view network delivery on the head-motion path but shifts compute, memory, and cache constraints to the headset.The contrast follows the stated mode-specific limitations.Source: B. VR Rendering and Streaming · p. 4
  • Simplified scene streaming sends a TSDF mesh and quantized Gaussian primitives for local headset rendering, with later updates limited to added, changed, or removed primitives.This supports fast viewpoint changes between network updates but requires more headset memory and compute; visual quality depends on local-cache size and age.Source: B. VR Rendering and Streaming · p. 4

C. Human-agent interaction

Human-agent interaction grounds speech and text in a shared scene model, combining dialogue with controller pointing or selection. A language-model router turns requests into typed scene operations, while validation and operator confirmation gate robot-directed actions.

C. Human-agent interaction

  • Pointing rays resolve spatial references to scene objects and persistent IDs, blending that grounding with dialogue context.The operator may communicate through speech, text, object selection, or spatial pointing.Source: C. Human-agent interaction · p. 4
  • A versioned symbolic Scene Model exposes object IDs, labels, centroids, bounding boxes, and reconstruction attributes as the interface between mapping and language grounding.It decouples language interpretation from the raw Gaussian representation.Source: C. Human-agent interaction · p. 4
  • Each interaction combines a request, optional spatial grounding, scene state, and interaction history; the agent returns either a clarification or a typed decision.A typed decision contains a decision kind, tool or mission name, and structured arguments.Source: C. Human-agent interaction · p. 4
  • A deterministic pre-router handles unambiguous requests and policy checks; a schema-constrained language-model router handles the rest, with validation before execution.Robot-directed actions are staged for operator confirmation.Source: C. Human-agent interaction · p. 4

D. Agent-robot interaction

The agent requests navigation or additional observations using scene geometry, robot pose, and mapping quality. Goals can target objects, coordinates, or regions; robot missions execute predefined tasks, with navigation requiring operator approval.

D. Agent-robot interaction

  • Goals may target objects, scene coordinates, or predefined regions, and the agent requests navigation or additional observations using mapping quality.Agent-robot interaction uses scene geometry and robot pose as well as mapping quality.Source: D. Agent-robot interaction · pp. 4–5
  • The robot backend runs predefined missions such as approach, inspection, reinspection, and patrol; navigation goals require operator approval.The agent tracks arrival, failure, and timeout outcomes.Source: D. Agent-robot interaction · p. 5
  • Registering a revisit does not establish that the robot moved or collected new data; execution feedback and incoming sensor observations verify those outcomes.Source: D. Agent-robot interaction · p. 5

IV. EVALUATION AND EXPERIMENTS

The experiments span reconstruction, VR, human-agent interaction, and agent-robot interaction across real-world platforms and recorded scenes. Agent routing is evaluated separately in a sandbox, so its results complement rather than replace physical-system evaluations.

A. Experimental Setup

  • The reconstruction setup used an i9-14900K desktop with an RTX 5090 GPU and a second identical computer for semantics; separating semantic processing was optional.The separate semantic server enabled a higher SAM3 segmentation rate and larger reconstructed scenes.Source: A. Experimental Setup · p. 5
  • Mobile trials used ZSL-1 and ZSM-1 quadrupeds with onboard localization and navigation, while reconstruction and agent inference ran on an external GPU workstation.The setup used 640 × 480 RGB-D acquisition at 30 Hz; navigation required operator approval and onboard safety checks.Source: A. Experimental Setup · p. 5
  • Additional reconstruction experiments covered a workbench-mounted UR10, humanoids tracked only by inline ORB-SLAM3, handheld RGB-D data, and a drone localized with Vicon markers.The drone dataset compared Vicon and ORB-SLAM3 reconstruction quality.Source: A. Experimental Setup · p. 5
  • The agent routing evaluation used 1,471 cases from 240 task specifications, 12 families, and four recorded scenes, with planned cases including transport errors and context overflows kept in the denominator.Recorded merge events, Replica referring expressions, metadata controls, and context and shared-GPU effects were also tested.Source: A. Experimental Setup · p. 5
  • The agent sandbox isolates the agent and map interface, so its results should be read alongside—not instead of—the reconstruction, VR, and live robot evaluations.It uses recorded scene states and an isolated in-process sandbox.Source: A. Experimental Setup · p. 5

B. Reconstruction Evaluation

CognitiveReality outperforms the Gaussian-plus-SDF baseline on the reported matched mapping sequences, with gains also reported across mapping, fusion, and semantic-segmentation evaluations. Semantic results depend on class coverage and segmentation-server capacity.

B. Reconstruction Evaluation

  • 26.54 vs. 22.86 dB on quadruped sequence A; on the manipulator, CognitiveReality reaches 23.66 vs. 22.44 dB and 24.23 vs. 22.83 dB in the quality and boost settings.Table I compares online mapping on the same sequences with matched settings.Source: B. Reconstruction Evaluation · pp. 5–6, Table I · p. 6
  • 21.91 vs. 16.11 dB and 19.54 vs. 11.44 dB on two quadrotor flights; 23.20 vs. 15.32 dB on the Tien Kung head camera within the 8 m fusion range.Source: B. Reconstruction Evaluation · p. 6
  • On the ZSL-1 camera, gyroscope gating raises PSNR from 18.09 to 20.70 dB, and the budget-calibrated schedule reaches 22.73 dB versus 22.89 dB for the vibration-free body camera.Uncertainty-weighted fusion raises sequence A from 26.45 to 27.00 dB, or 27.35 dB with third-order spherical harmonics; Replica office0 PSNR remains 38.95 dB with or without instance semantics.Source: B. Reconstruction Evaluation · p. 6
  • CognitiveReality runs at 51–66 frames per second on 720p sequences, 132–158 on VGA flights, and 29–42 on the 20 Hz manipulator input; the baseline is faster on most sequences.The baseline uses all frames in RAM and omits visual tracking and noise-weighted fusion.Source: B. Reconstruction Evaluation · p. 6
  • With closed vocabulary, CognitiveReality is near HoloAgent-Memory in mIoU (29.5 vs. 29.9) and higher by 3.0, 8.7, and 12.9 points in the other reported metrics.Table II reports zero-shot 3D segmentation on Replica, with IoU and accuracy averaged over classes or weighted by class frequency.Source: B. Reconstruction Evaluation · p. 6, Table II · p. 6
  • At 2 Hz segmentation, the 20 fps stream’s result is unchanged at 29.2; at 3 Hz, dropped requests cost 2 mIoU points, while the mapper never drops frames.Growing the vocabulary from 11 generic classes loses 12 mIoU with the default prompt; a structure-aware inventory prompt recovers 5 points.Source: B. Reconstruction Evaluation · p. 6

C. VR Evaluation

The VR evaluation compares remote stereo image streaming with simplified scene streaming and local rendering. The latter removes per-view network delivery from the head-motion path, but trades local compute, memory, and bounded scene fidelity for network-independent viewpoint updates.

RSIS

  • RSIS produces 54–84 host-rendered fps with 25–46 ms median request-to-decode latency, while requiring little scene-side computation or memory on the headset.As the Gaussian map grows, rendering cost increases on the host.Source: C. VR Evaluation · p. 6

3SLR

  • 3SLR reaches 66–85 fps with approximately 9–10 ms p95 GPU frame time and up to 36.7 dB PC-HS PSNR, while supporting up to 200k resident Gaussians.It streams a TSDF mesh and quantized Gaussian updates; the headset renders from the latest OpenXR pose, with only changed primitives transmitted after initial transfer.Source: C. VR Evaluation · p. 7

Delivery-mode comparison

  • RSIS preserves the full host representation but remains sensitive to rendering and network latency; 3SLR trades local compute, memory, and bounded scene fidelity for network-independent viewpoint updates.The reported timing values characterize different pipeline stages and are not a direct motion-to-photon comparison.Source: C. VR Evaluation · p. 7

D. Human-Agent Interaction

The human-agent evaluation tests typed command routing, object-reference validity, grounding, metadata use, and resource costs. The deployed local router outperforms rules-only routing, while the other tests characterize where scene grounding and the operating envelope remain consequential.

Routing and calls

  • Tool exact match rises from 48.33% to 81.24% and full-call correctness from 42.96% to 72.88% over rules-only routing across 1,471 planned cases.Effect correctness is 84.70%; post-validation adds 3.04 percentage points in paired full-call correctness, and the complete guard path produces no unsafe proposals in 71 sandbox policy cases, not a measure of physical safety.Source: D. Human-Agent Interaction · p. 7

Temporal replay

  • With causally available merge evidence, replay completes 117 of 167 episodes and correctly redirects 101 absorbed identifiers, while confirmation-time revalidation remains a focused extension.The confirmation gate forwards the resolved reference without rechecking it against a later map update.Source: D. Human-Agent Interaction · p. 7

Referring tests

  • Typed-agent target-ID correctness is 21.5/28.1% and Acc@0.25 is 13.6/30.3% on office0/office1; with the ground-truth object list, language-model target-ID correctness reaches 61.6/71.9%.The predicate oracle reaches 98.3/97.8%; ID and overlap are reported separately because online boxes represent partial observations, whereas Replica boxes cover complete mesh geometry.Source: D. Human-Agent Interaction · p. 7

Metadata controls

  • Named-target agreement rises from 24.7% with images alone to 95.9% with image and metadata, while metadata alone reaches 99.3%.Under conflicting metadata, the model repeats the supplied class in 91.8% of 146 cases and acknowledges the conflict in 18.5%; agreement is not independent visual-recognition accuracy.Source: D. Human-Agent Interaction · p. 7, Table IV · p. 7

Context and resources

  • Compact context preserves 72/100 and 70/100 full calls on matched office0 and lab1 subsets, while whole-map lab1 falls to 42/100 with 68 context overflows.The results support selective scene serialization rather than removing memory fields, since execution still uses the complete state.Source: D. Human-Agent Interaction · p. 7
  • In a shared-GPU pilot, 12 commands/min increases Gaussian round time by 12.7% and peak VRAM by 3,294 MB, with no frame gaps.A separate 362-request live trace measures 1,422/1,993 ms p50/p90 push-to-talk latency; these results establish the tested operating envelope, not a general real-time guarantee.Source: D. Human-Agent Interaction · p. 7

E. Agent-Robot Interaction

Live trials connect agent requests to robot motion and map updates: most marked navigation goals were reached, and all re-observation requests improved object rendering quality. The conclusion emphasizes persistent object references and reports controlled-agent and live-deployment results.

E. Agent-Robot Interaction

  • 26 of 30 navigation requests reached the marked goal, and all 20 re-observation requests were executed, raising object rendering PSNR by 2–5 dB depending on the frame.Counting only pipeline-attributed navigation failures, 26 of 29 requests (0.90) reached the goal; no request produced motion without confirmation.Source: E. Agent-Robot Interaction · p. 8

V. CONCLUSION

  • CognitiveReality combines online Gaussian-SDF reconstruction, a versioned semantic Scene Model, typed agent tools, and an immersive operator interface.Persistent object references connect language, pointing, visualization, and robot-directed requests while allowing failure modes to be evaluated separately.Source: §V · p. 8
  • The controlled evaluation improves tool exact match from 48.33% to 81.24% and full-call correctness from 42.96% to 72.88% over rules-only routing; replay correctly redirects 101 absorbed identifiers.Confirmation-time revalidation remains a focused extension, and metadata agreement should not be interpreted as independent visual recognition.Source: §V · p. 8
  • Live trials across multiple platforms and localization sources reached 26 of 30 marked navigation goals and completed all 20 re-observation requests.No navigation request produced motion without operator confirmation.Source: §V · p. 8
Loading 2609.31418v1…