Source-linked AI summary
Harness Robotic OS: A Unified Embodied-Agent Runtime for Closed-Loop Quadruped Inspection
Yaoyuan Yan, Zhiyou Heng, Haoxiang Jie, Gang Liu, Hongjie Yan, Wei Zhou
TL;DR
Autonomous property inspection needs a traceable loop connecting heterogeneous sensing, autonomy, scene understanding, human interaction, and enterprise response. HROS provides that governed embodied-agent runtime, and Argos realizes it for residential inspection, achieving complete waypoint reachability, sub-10 cm outdoor localization error, sub-200 ms obstacle response, 85–95% representative hazard detection, and 99% communication/reporting success.
Problem
Existing inspection implementations fragment sensing, autonomy, reasoning, interaction, and operations across task-specific interfaces, limiting shared context, experience reuse, and safely controlled adaptation.
Method
HROS unifies robot resources, reusable autonomy skills, shared context, multimodal reasoning, voice interaction, hierarchical memory, and safety-gated versioned adaptation, instantiated by Argos on a Vbot quadruped.
Results
Argos achieved 100% waypoint reachability, outdoor localization error below 10 cm, obstacle-response latency below 200 ms, representative hazard-detection rates of 85–95%, and 99% communication and reporting success.
Takeaways & Limitations
The results validate the deployed navigation and inspection closed loop, while HROS provides an extensible foundation for voice-aware, memory-augmented, and governed self-evolving agents.
Takeaways & Limitations
Long-term map maintenance, open-world hazard recognition, agent evaluation and safety, and human–robot collaboration remain future-work requirements.
Abstract
from arXiv · showhide
Autonomous property inspection requires more than robust robot navigation: a deployable system must connect heterogeneous sensing, reusable autonomy capabilities, multimodal scene understanding, human interaction, and enterprise response within a traceable operational loop. Existing quadruped inspection systems commonly integrate these functions through task-specific interfaces, making contextual coordination, knowledge reuse, and controlled adaptation difficult. This paper presents \textit{Harness Robotic OS} (HROS), a unified embodied-agent runtime, and Argos, its realization for residential-community inspection. HROS organizes the system into robot runtime, embodied autonomy skills, cognitive agent runtime, and interaction and operations planes. A shared context connects physical state with agent reasoning; streaming ASR/TTS supports voice-based mission interaction; hierarchical working, episodic, and semantic memory preserves operational knowledge; and a safety-gated self-evolution loop converts execution traces into versioned candidate updates without permitting unconstrained online modification. The Argos prototype integrates a Vbot quadruped, Fast-LIO2 localization and mapping, Hobot-Stereo depth perception, PCT-Planner global planning, EGO-Planner local motion generation, and OpenClaw-orchestrated Qwen3-VL inspection analysis. Experiments in a residential property environment achieved 100\% waypoint reachability, outdoor localization error below 10~cm, local obstacle-response latency below 200~ms, representative hazard-detection rates of 85--95\%, and 99\% success in alarm delivery and structured-report generation. These results validate the deployed navigation and inspection closed loop, while HROS provides an extensible software foundation for memory-augmented, voice-aware, and continuously improvable embodied inspection agents.
1 Introduction
Residential inspection requires repeatable physical coverage and auditable operational outcomes across changing, hazard-rich environments. HROS addresses fragmented implementations by unifying sensing, autonomy, reasoning, interaction, memory, and governed adaptation, while Argos realizes and evaluates this loop on a quadruped.
- Inspection setting: Residential inspection spans broad public areas with changing illumination, people, vehicles, temporary obstacles, and diverse safety and sanitation risks.Manual patrols provide contextual judgment, but frequency, consistency, and traceability are constrained by labor and individual experience.
- System gap and contribution: Fragmented task-specific interfaces disconnect semantic intent from execution state, limit experience reuse, and complicate safe evaluation, tracing, and rollback.These limitations are especially relevant to repeated inspection with on-site and remote collaborators.
- System gap and contribution: HROS unifies sensing, autonomy, reasoning, interaction, and operational feedback within one governed embodied-agent loop.The runtime maintains shared physical-cognitive context, reusable skills, multimodal orchestration, and safety-gated updates.
- System gap and contribution: HROS combines voice-aware interaction, hierarchical memory, reusable embodied skills, and safety-gated self-evolution for traceable inspection operations.Its contributions include grounded ASR/TTS, working–episodic–semantic memory, versioned rollout, provenance, and rollback.
- Argos realization: Argos integrates Fast-LIO2, Hobot-Stereo, PCT-Planner, EGO-Planner, and OpenClaw-orchestrated Qwen3-VL for residential-community inspection.Inspection outputs are grounded in time and location before structured reports and enterprise alerts are generated.
X Inspection Mission Console
The inspection mission console is presented as an interaction-and-cognition layer combining voice and multimodal I/O with shared context, persistent memory, and governed adaptation.
- Inspection Mission Console: The architecture positions shared context and persistent memory alongside streaming ASR, TTS, and multimodal inputs for embodied-agent operation.The supplied architecture labels identify these as core cognitive-runtime and interaction capabilities.
- Inspection Mission Console: The cognitive runtime combines voice and multimodal interaction, hierarchical agent memory, reusable embodied skills, and safety-gated self-evolution.Execution experience is transformed into validated runtime updates.
2 Related Work
Related work spans localization, global and local planning, vision-language inspection, and agent frameworks. HROS addresses the integration gap by placing agent orchestration above deterministic autonomy while preserving explicit context, memory, and safety gates.
- Localization and Mapping: Fast-LIO2 fuses LiDAR and IMU information for high-precision state estimation with relatively low computational overhead and reduced single-sensor drift.The paper adopts it for localization and mapping.
- Global Planning: Global planning research includes graph search, sampling-based, and kinodynamic methods, with trade-offs involving optimality, efficiency, smoothness, convergence, and dynamic constraints.The cited survey passage places Dijkstra, A*, D*, Theta*, JPS, RRT-family, Hybrid A*, and related methods in this landscape.
- Local Planning: Local planning has progressed from reactive avoidance toward continuous trajectory optimization and dynamic-environment prediction, with real-time performance and computational cost remaining trade-offs.The discussion contrasts DWA, TEB, MPC, MPPI, and related approaches.
- Local Planning: EGO-Planner avoids explicit ESDF construction by using local obstacle-distance information, reducing computational overhead while maintaining smooth and dynamically feasible trajectories.The paper adopts it for local path planning because of its real-time performance and robustness.
- Multimodal Inspection and Embodied Agents: Vision-language models support open-vocabulary, instruction-conditioned scene interpretation, while generic agent loops leave mobile-robot timing, safety, state-consistency, and recovery unresolved.These observations motivate HROS’s integration of multimodal reasoning with governed embodied autonomy.
- Embodied-agent integration: HROS separates agent orchestration from deterministic localization, perception, and motion skills to address timing, safety, state consistency, and recovery.Candidate adaptations must pass an offline safety gate, preserving real-time autonomy guarantees while enabling semantic and operational improvement.
3 Harness Robotic OS Architecture
HROS connects robot hardware, reusable autonomy skills, agent orchestration, shared context, memory, multimodal reasoning, voice interaction, and property-management workflows in a governed closed-loop runtime. Argos implements this architecture through deterministic navigation and perception pipelines plus schema-constrained inspection reporting and enterprise delivery.
- Harness Robotic OS Architecture: HROS composes sensing, localization, planning, inspection understanding, and operational feedback as traceable capabilities rather than isolated modules.Its self-evolution path evaluates execution experience and promotes only validated adaptations into the deployed skill runtime.
- Robot Runtime: The Vbot quadruped supplies sensing, computation, communication, and locomotion for traversal across ramps, thresholds, speed bumps, uneven terrain, and cluttered passages.Its platform includes stereo cameras, 16-line LiDAR, IMU, GNSS, cellular connectivity, and an RDK S100P edge computer.
- Embodied Autonomy Skills: Fast-LIO2 builds the prior map and estimates six-degree-of-freedom pose, while Hobot-Stereo supplies near-field depth for low or partially occluded obstacles.The fused representation is converted into the planning stack’s obstacle format.
- Embodied Autonomy Skills: PCT-Planner preserves mission-level waypoint coverage while EGO-Planner continuously optimizes dynamically feasible local trajectories around immediate and temporary obstacles.Each capability reports timestamp, execution state, and confidence or status for runtime coordination.
- Voice Interaction: Streaming ASR grounds spoken instructions in robot pose, mission, scene, and access policy, while TTS reports acceptance, progress, alerts, and recovery information.Ambiguous or safety-critical instructions require explicit confirmation.
- Hierarchical Memory: Working, episodic, and semantic memory retain current context, time-indexed mission experience, and stable site knowledge respectively.The three levels cover short-horizon state, inspection trajectories and outcomes, and persistent maps, assets, rules, and defects.
- Governed Self-Evolution: Self-evolution converts execution traces and human feedback into candidate memory, prompt, tool-policy, task-graph, or skill updates that undergo evaluation before deployment.The process is governed rather than an unconstrained online model modification.
- Multimodal Inspection: OpenClaw binds inspection images to pose, time, waypoint, mission, and policy before Qwen3-VL produces schema-constrained hazard events.Only schema-valid events enter reporting and alerting, while original evidence and model responses preserve auditability.
4 System Implementation
Argos combines map-based localization, coverage-oriented global planning, dense stereo–LiDAR obstacle perception, local replanning, and evidence-preserving inspection reporting. These components share mission and execution context so navigation outputs remain connected to semantic analysis and operational delivery.
- 4.1 LiDAR–Inertial Mapping and Relocalization: Fast-LIO2 supports mapping, online localization, and relocalization in a shared frame used by planners and inspection-event logging.The shared frame also binds images, waypoints, hazards, and reports for spatial memory retrieval and cross-mission comparison.
- 4.2 Global Inspection Planning: PCT-Planner connects policy-defined inspection viewpoints into reusable coverage routes, while runtime state tracks active segments, waypoints, and remaining actions.The agent can pause, resume, or reorder non-safety-critical tasks without modifying the local controller.
- 4.3 Stereo–LiDAR Obstacle Perception: Hobot-Stereo supplies dense near-field geometry that complements LiDAR’s wider-range metric structure for local obstacle perception.Depth is filtered by range and confidence, transformed into the map frame, and conservatively expired when not re-observed.
- 4.3 Stereo–LiDAR Obstacle Perception: EGO-Planner optimizes dynamically feasible local trajectories from the global route, robot state, and fused obstacle representation, including detours around temporary objects.Its output is converted into quadruped-controller commands subject to velocity, clearance, and continuity limits.
- 4.4 Inspection Analysis and Reporting: OpenClaw binds inspection images to pose and mission context, validates Qwen3-VL outputs against structured event schemas, and routes accepted evidence to enterprise workflows.Original inferences and human corrections remain preserved in the operational record.
5 Deployment and Evaluation Protocol
The deployment protocol evaluates Argos as a traceable five-stage lifecycle, from map commissioning and route execution through contextualized inspection analysis and enterprise reporting. Controlled trials additionally test voice interaction, hierarchical memory, and governed adaptation using reproducibility-oriented metrics.
- Deployment lifecycle: The field evaluation follows a five-stage lifecycle that tests state and evidence transfer across the HROS runtime, not only individual navigation components.This protocol reflects the same lifecycle used in routine deployment.
- 5.1 Phase 1: Establish 3D Point Cloud Map: During commissioning, synchronized LiDAR–IMU data builds a reviewed prior map, after which PCT-Planner forms a versioned route through eight required inspection waypoints.The stored mission template includes route and waypoint actions.
- 5.2 Autonomous Mission Execution: During autonomous execution, Fast-LIO2 localizes the robot while EGO-Planner updates trajectories from fused LiDAR–stereo obstacles and records waypoint transitions, replanning, failures, and recovery.This preserves execution traces for later operational analysis.
- 5.3 Inspection Analysis and Reporting: OpenClaw contextualizes traversal and 360° inspection images for Qwen3-VL, parses responses into event schemas, and validates them before operational use.Validated events retain timestamps, map locations, region labels, source evidence, delivery acknowledgments, and operator corrections.
- 5.6 Controlled Evaluation of Cognitive Runtime: Controlled experiments measure voice command understanding, hierarchical-memory retrieval, and governed adaptation with task, accuracy, latency, completion, and reproducibility metrics.The protocol distinguishes transcription quality from mission-level command understanding and requires counts, evaluation splits, repetitions, medians, and 95th percentiles.
6 Experimental Results
The residential deployment achieved reliable navigation and operational delivery across the tested inspection loop, while hazard-recognition performance varied by visual class. The authors characterize these results as feasibility evidence and identify broader, longer-term evaluation as future work.
- 6.1 Navigation and Mission Execution: 100% of configured waypoints were reached, outdoor localization error stayed below 10 cm, and obstacle-response latency remained below 200 ms.Endurance exceeded 3 h, while a full-coverage mission required no more than 60 min under the evaluated configuration.
- 6.2 Hazard Recognition and Operational Delivery: Hazard detection ranged from 85% to 95% across visual classes, with false-positive and missed-detection rates each below 5%.Garbage overflow and fire-passage obstruction reached 95%, while public-facility damage reached 85%.
- 6.2 Hazard Recognition and Operational Delivery: 99% of communication and reporting operations succeeded, transferring most accepted events into the enterprise workflow without loss.The workflow connects accepted inspection events to downstream operational delivery.
- Overall findings: The reported results establish feasibility across navigation, semantic inspection, and operational delivery for the deployed Argos pipeline.Future evaluations will examine additional properties, longer operating periods, memory, voice interaction, and governed adaptation.
7 Discussion
HROS separates deterministic autonomy from cognitive adaptation while linking mission reasoning to robot state and operational outcomes. Its shared context, provenance tracking, hierarchical memory, and governed evolution support actionable, auditable inspection without allowing reasoning failures to bypass motion control.
- Separating deterministic autonomy from cognitive runtime prevents reasoning failures from directly bypassing motion control while keeping navigation events available to inspection workflows.
- Binding images and model outputs to map frames, missions, and inspection policies turns observations into structured events for review and work-order handling.
- Hierarchical memory retains active execution, mission history, and stable site knowledge across different horizons for recurring patrols.
- Candidate changes to memory, prompts, tools, or task graphs require offline evaluation, provenance tracking, versioned rollout, and rollback before deployment.
8 Limitations and Future Work
The current field system demonstrates an integrated inspection pipeline but remains bounded by unresolved challenges in map maintenance, hazard recognition, agent evaluation, safety, and human–robot collaboration.
- Long-term deployment must handle parking, construction, vegetation, and seasonal changes through incremental mapping, change detection, and multi-session map management.
- Open-world hazard recognition requires more diverse labeled data, calibrated confidence, policy constraints, and systematic handling of ambiguous observations.
- Memory and self-evolution require benchmarks for retrieval quality, adaptation benefit, regression risk, and rollback reliability before autonomous updates enter production.
- Future human–robot evaluations should cover outdoor-noise ASR robustness, safety-critical confirmation design, operator workload, and enterprise-system integration.
9 Conclusion
The paper presents HROS as a unified runtime and Argos as its quadruped inspection realization. Field results demonstrate feasibility in a residential property environment, while governed adaptation and shared memory define a path toward broader evaluation.
- Field trials achieved complete waypoint reachability, sub-10-cm outdoor localization error, sub-200-ms obstacle response, and high delivery reliability for structured events.
- HROS unifies robot resources, autonomy skills, multimodal reasoning, memory, human interaction, and enterprise operations with Argos as its quadruped realization.
- Shared context and hierarchical memory connect repeated missions, while safety-gated evolution supports improvement without uncontrolled online modification.
- Future work will evaluate the system across sites and quantify cognitive components, long-term memory, voice interaction, and governed adaptation under operational noise and change.