ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

Motivation

• The introduction identifies reasoning-execution, embodiment-generalization, and persistent embodied-memory gaps, alongside the need for benchmarks testing executable multi-scene scenarios with dynamic events and trace-grounded scoring.It notes that many systems lack an intermediate layer for decomposition, tools, skill delegation, verification, and recovery, while memory must preserve multi-modal, relational, source-grounded experience.Source: §1 · p. 4

A General Robotic Agent OS Architecture Bridging VLMs and Physical Execution

• ABot-AgentOS places a modular, plugin-based Agent OS above VLMs and low-level controllers, linking reasoning to dialogue, locomotion, manipulation, and multi-stage verification across robot embodiments.Its components include a main LLM, context management, a skill runner, and tool interfaces; embodiment-specific mobility and manipulation interfaces support heterogeneous robots.Source: §1 · p. 5

A Comprehensive Benchmark for Agent Capabilities

• EmbodiedWorldBench evaluates long-horizon embodied capabilities across executable indoor, outdoor, and hybrid scenarios, including planning, tool use, recovery, interaction, and partial observability.The benchmark is designed to move beyond isolated manipulation or navigation-only evaluation.Source: §1 · p. 5

A Universal Multi-modal Memory System with Trace-Driven Lifelong Self-Evolution

• ABot-AgentOS stores embodied experience in a typed, source-grounded multi-modal graph and uses retrieval traces to diagnose memory failures and promote gated fixes only to later evaluation splits.The graph represents entities, events, places, sessions, visual evidence, temporal context, spatial relations, and provenance; evo-assets target failures including evidence selection, temporal grounding, entity matching, and answer composition.Source: §1 · p. 5

Deployment-oriented small-model training pipeline

• A deployment-oriented pipeline distills long-horizon tool use from stronger teacher agents into a smaller embodied model using supervised fine-tuning, reinforcement learning, and a self-evolving LLM-as-a-Judge reward engine.The authors focus on the method and do not release private data or production results for this component.Source: §1 · p. 5

2.1 Architecture Overview

• ABot-AgentOS adds a unified cognitive layer above robot hardware and low-level controllers, connecting perception, memory, reasoning, planning, and execution across heterogeneous embodiments.It is designed to work with humanoids, quadruped robot dogs, mobile manipulators, and robotic arms rather than replacing their existing control stacks.Source: §2.1 · p. 5
• A Tiny LLM handles every interaction turn on the edge, while a cloud-based Large LLM is invoked on demand for complex reasoning, long-horizon planning, or ambiguity resolution.Multi-modal inputs from microphones, apps, cameras, and other sensors enter a perception–reasoning–action runtime loop, which dispatches outputs to robot hardware.Source: §2.1 · p. 5, Figure 1 · p. 6
• The Agent Harness organizes prompts, tools, APIs, and execution protocols into a reliable loop using verification-aware ReAct, context management, and skill evolvement.Context management selects, compresses, and retrieves relevant observations, dialogue, robot states, and memory; skill evolvement refines and expands skills through physical-world interaction.Source: §2.1 · p. 6
• A unified Skills and Tools layer abstracts manipulation skills [101], navigation skills [18] [99], motion control, vision, and extensible tools for reuse across robot embodiments.This lets different platforms reuse high-level reasoning and task-planning logic.Source: §2.1 · p. 6
• The multi-modal memory system separates robot-specific edge-side private memory from cloud-side common memory that supports knowledge transfer across robots.Private memory includes maps, semantic knowledge, interaction history, preferences, and local experiences; common memory aggregates shareable maps and task experiences through upload and download.Source: §2.1 · p. 6, Figure 1 · p. 6
• The architecture integrates edge intelligence, cloud reasoning, multi-modal memory, and modular skill execution while decoupling high-level cognition from specific hardware embodiments.This provides a scalable foundation for agents that perceive, remember, reason, act, and evolve in real-world environments.Source: §2.1 · p. 6

2.2 Agent Harness

• The Agent Harness separates planning, procedural execution, and verification in a closed loop to ground reasoning in environment facts and reduce procedural drift and premature termination.The main LLM plans with memory and context, the Skill Runner executes delegated subtasks, and the Verifier checks progress and completion rather than relying on language-level belief alone.Source: §2.2 · p. 7

2.2.1 Scene-Conditioned Task Planning

• Scene-conditioned planning maps instructions to revisable high-level plans and observable completion conditions using the current scene, memory, robot state, history, and available skills.The main LLM updates plans from new observations, tool results, skill summaries, and verifier feedback, deciding when to use tools directly, delegate, or finish.Source: §2.2.1 · p. 7

2.2.2 Skill Runner for Procedural Execution

• The Skill Runner executes sustained subtasks in an isolated local context, managing observations, state, failed attempts, and recovery while returning compressed outcomes to the main LLM.It maintains procedural continuity and reports whether the subgoal succeeded, why it failed, discovered scene information, recovery attempted, and how to use the outcome.Source: §2.2.2 · p. 8

2.2.3 Multi-Stage Verification

• Multi-stage verification checks environment-grounded progress at runtime, delegated-skill success, and attempted task termination, reducing stagnation, misjudgment, and premature termination.The Verifier evaluates trajectories and skill states, semantic subgoal completion, and whether the instruction is supported by execution history and observations before allowing the task to finish.Source: §2.2.3 · p. 8

2.2.4 Edge-Cloud Collaborative Routing

• Edge-cloud routing lets an on-device model handle routine requests locally or escalate complex tasks, using a learned policy informed by training samples and execution feedback.The policy also identifies when additional observation is needed before routing, balancing latency and deployment cost with cloud-scale reasoning for tasks requiring it.Source: §2.2.4 · p. 8

2.3 Memory System

• The persistent memory substrate converts multimodal experience into compact, typed, source-grounded records for traceable retrieval across sessions.Records cover observations, dialogue, semantic facts, visual evidence, identities, object states, spatial relations, temporal context, and task traces.Source: §2.3 · p. 9

2.3.1 Graph Memory Representation

• Unlike text-chunk retrieval, graph retrieval expands relevant seed nodes into evidence subgraphs for reasoning over identity, time, location, provenance, and spatial relations.The platform-agnostic graph encodes entities, evidence units, places, sessions, events, and typed relations, with provenance and temporal fields linking records to their sources.Source: §2.3.1 · p. 10

2.3.2 Multi-modal Memory Updating

• Multi-modal updating normalizes inputs into a shared graph schema, selectively storing useful semantic records while maintaining compact, auditable memory over time.The writer merges compatible near-duplicates, bounds representative evidence for frequently observed entities, and supersedes stale facts through temporal edges rather than deleting them.Source: §2.3.2 · pp. 10–11

2.3.3 Retrieval and Grounded Answering

• Retrieval combines semantic, lexical, metadata, and node-type cues, then expands top-ranked seeds along typed edges to provide a compact evidence context and trace.Answers are grounded in retrieved evidence; traces record retrieved nodes and edges, sources, summaries, retrieval stages, and ranking signals for inspection and failure analysis.Source: §2.3.3 · p. 11

2.3.4 Failure-Driven Lifelong Self-Evolution

• Failure-driven self-evolution diagnoses traces, proposes constrained evo-assets, and promotes accepted improvements only to later evaluation splits, preventing same-split leakage.Candidate assets undergo target and regression checks; accepted assets are versioned JSON DSL records rather than executable generated Python code.Source: §2.3.4 · pp. 12–13

2.3.5 Edge-Cloud Collaborative Memory Management

• Over 99% accuracy in identifying whether memory items are suitable for cloud synchronization enables public knowledge sharing while reducing sensitive-information uploads.Private-by-default edge memory retains locally perceived data; only explicitly shareable public environmental information enters common cloud memory.Source: §2.3.5 · p. 13

3.1 Overview

• Each case is an executable scenario that jointly tests multiple capabilities, with trace-grounded, auditable outcomes and reproducible trajectories under the same configuration.A representative episode traverses streets, a backyard, and an indoor living room while completing four sub-goals and reporting results.Source: §3.1 · p. 14

3.2.1 Environment and Semantic Map Annotation

• UnrealZoo supplies photo-realistic UE5 environments, which annotators map by marking reachable areas, key objects, points of interest, interaction targets, and navigation paths.The resulting semantic maps support task design and query generation.Source: §3.2.1 · pp. 14–15

3 EmbodiedWorldBench

• Across 16 scenes, four difficulty levels, and over 200 tasks, EmbodiedWorldBench evaluates compound activities spanning indoor and outdoor spaces, including navigation, NPC interaction, and environment perception.Tasks also cover object search, dynamic instruction response, and state tracking, with agents receiving only a filtered semantic map and relying on autonomous perception and reasoning.Source: §3 · p. 14, Figure 5 · p. 15

3.2.2 Query Generation

• Task generation normalizes maps into point, room, and polygon layers, then creates scene-grounded tasks with human review, four calibrated difficulty levels, and automated checks for visibility isolation and consistency.The framework covers 16 scenes and over 300 annotated waypoints; tasks conceal NPC positions, evaluation signals, and expected trajectories from agents.Source: §3.2.2 · pp. 15–16

3.3.1 Embodied Evaluation Workflow

• Validated queries run as closed-loop episodes in which agents receive only instructions and filtered maps, while structured execution traces support deterministic checks and evidence-grounded evaluation.NavMesh navigation abstracts low-level movement so the agent can focus on high-level decisions, and dynamic events can require replanning.Source: §3.3.1 · p. 16

3.3.2 Evaluation Metrics

• Task Success Rate requires all subtasks and terminal conditions to be met, while Goal Completion Rate measures the proportion of completed subtasks.Geometric criteria are checked programmatically; semantic criteria are judged from run-derived evidence, with structured verdicts that do not rely on direct environmental access or hidden evaluator states.Source: §3.3.2 · pp. 16–17

4.1 Overview

• The pipeline connects text-based environment construction, teacher trajectory distillation, supervised fine-tuning, and reinforcement learning into a closed loop.Teacher rollouts can be filtered into SFT examples, while student rollouts in the same sandbox supply evidence for LLM-as-a-Judge rewards during online RL.Source: §4.1 · p. 17

4.2 Text-Based Environment and Task Construction

• The sandbox turns language tasks into interactive worlds where validated skill calls, obstacles, human assistance, and state updates train replanning and recovery.Easy, Medium, and Hard variants form a curriculum from basic decomposition and skill selection to multi-turn interaction, persistent goal tracking, and recovery from nested failures.Source: §4.2 · p. 18

4.3 Teacher Trajectory Distillation and SFT

• Teacher traces preserve reasoning, tool calls, observations, help-seeking, failures, retries, and recovery, and a judge filters them before conversion into structured tool-call SFT examples.The student learns decomposition, tool selection, recovery, help seeking, multi-turn goal maintenance, and avoidance of unsupported actions, providing the prior for online RL.Source: §4.3 · p. 18

4.4 RL: Online Exploration with LLM-as-a-Judge Reward

• Online RL trains only the SFT-initialized policy while sandbox, NPC, and frozen LLM-as-a-Judge components provide feedback; GiGPO combines episode- and step-relative advantages.Multiple rollouts from the same initial state are compared to favor better complete solutions and local actions in comparable situations.Source: §4.4 · pp. 18–19

4.5 Self-Evolving Reward Engine

• The self-evolving reward engine links trajectory reward generation, Meta-Judge reliability validation, and multi-agent prompt updates as skills, tasks, and behaviors change.It serves both online RL and ongoing reward-prompt improvement.Source: §4.5 · p. 19

4.5.1 Reward from LLM-as-a-Judge

• Skill-aware turn rewards combine LLM judgment with verifiable rule checks and omission penalties, while episode rewards assess efficiency, consistency, and completeness.Turn-level signals distinguish useful, redundant, unsafe, malformed, or context-inconsistent actions; episode-level feedback evaluates trajectory-wide task completion.Source: §4.5.1 · p. 19

4.5.2 Meta-Judge Validation

• Meta-Judge checks whether a first-order judge’s score and rationale are reliable against trajectory evidence and rubric criteria, producing diagnostic low-quality cases for refinement.Its dimensions are accuracy, logical soundness, completeness, clarity, and feedback value; cases identify underspecified, ambiguous, or embodiment-misaligned reward prompts.Source: §4.5.2 · pp. 19–20

4.5.3 Multi-Agent Self-Evolution

• Human alignment rises from roughly 60% for the initial Judge Model to above 90% with Meta-Judge-driven self-evolution.The multi-agent workflow clusters low-quality cases, analyzes defective rubric components, applies localized edits, and validates updates on the full set, rolling back regressions in overall quality, per-skill behavior, or Meta-Judge quality.Source: §4.5.3 · p. 20

5.1.2 Experimental Results

• With Qwen3.6-Plus, Ours improves TSR by 11.99% and GCR by 10.84% over the single-controller baseline; DeepSeek-V4-Pro (Ours) adds 6.22% TSR and 5.83% GCR.The comparison uses the same Qwen3-VL-Plus observation tool; the subset results are intended as initial validation, not a complete leaderboard.Source: §5.1.1 · pp. 20–21, §5.1.2 · p. 21

5.1.3 Analysis

• The agent’s simultaneous TSR and GCR gains indicate more trajectories reach complete success, with structured task state, delegated skill execution, and verification addressing context drift, premature termination, weak evidence gathering, and search loops.Remaining failures include coarse observation feedback, confusion between people and objects, and misjudging location from visible connected regions; uncertain recorder execution also complicates evaluation.Source: §5.1.3 · pp. 21–22

5.2.2 Experimental Results

• ABot-AgentOS Static scores 87.5 overall on LoCoMo, 59.9 on OpenEQA EM-EQA with 24 frames, and 88.6 overall on Mem-Gallery.On LoCoMo it exceeds reproduced Mem0 by 1.9 points; the OpenEQA figure is reported as up to 59.9, while Mem-Gallery remains below full-context upper bounds.Source: Table 2 · p. 22, Table 3 · p. 23, §5.2.2 · p. 24

5.2 Memory Evaluation

• The memory evaluation fixes the base hybrid graph retriever and reports dataset-specific results across LoCoMo, OpenEQA, Mem-Gallery, NExT-QA, and EgoLife rather than averaging scores.Standard memory graphs use only inference-time information, excluding gold answers, rationales, and supporting evidence.Source: §5.2 · p. 22, §5.2.1 · p. 23

5.2.2 Experimental Results

• ABot-AgentOS Static reaches 76.5 Acc@All on NExT-QA validation, 3.2 points above GraphVideoAgent; separately, ABot-AgentOS-Qwen3.5-Flash averages 65.4% on EgoLifeQA, retrieving one relevant frame from 1FPS-sampled video.Source: §5.2.2 · p. 24

5.2.3 Lifelong Self-Evolution Results

• Self-evolution keeps the memory schema, retriever, and base writer–answerer fixed; post-evaluation failure diagnoses can promote gated runtime policies only for later splits.Ground-truth answers serve only as post-hoc correctness signals: they are unavailable at inference, are not written into memory, and cannot generate assets for the evaluated split.Source: §5.2.3 · p. 26, Figure 7 · p. 26
• Across five benchmarks, self-evolution improves primary scores: NExT-QA Acc@All +4.1, LoCoMo and OpenEQA +1.2 each, Mem-Gallery +0.4, and EgoLife average accuracy from 65.4 to 66.2.The largest category-level gains are on Mem-Gallery knowledge resolution, conflict detection, and multi-entity reasoning, and on EgoLife TaskMaster.Source: Figure 8 · p. 27, §5.2.3 · p. 27

5.2.4 Discussion

• Across datasets, the discussion attributes performance to structured, source-grounded evidence and reports reusable self-evolution gains without changing the retriever or leaking current-split answers.It identifies temporal normalization, scene disambiguation, relation and conflict handling, and causal-temporal reasoning as recurring improvement targets; environmental outcomes or human feedback could supply correctness signals in deployment.Source: §5.2.4 · p. 28

• ABot-AgentOS connects foundation VLM/VLA models to physical execution through a modular Agent Harness, typed source-grounded graph memory, and EmbodiedWorldBench for long-horizon embodied tasks.The benchmark spans indoor, outdoor, and hybrid scenarios with dynamic events, NPC interaction, and deterministic scoring.Source: §6 · p. 28
• Graph memory performs strongly across five memory and reasoning tasks, self-evolution improves later splits, and initial EmbodiedWorldBench evaluations suggest better long-horizon execution from hierarchical control, task memory, skill feedback, and finish-time verification.Self-evolution uses gated, trace-diagnosed evo-assets; the agent evaluation covers only an EmbodiedWorldBench subset.Source: §6 · p. 28
• The paper outlines transferring ABot-AgentOS-style tool use to smaller agent models through sandboxed trajectory distillation, SFT/RL, and judge-guided optimization.Source: §6 · p. 28
• Large-scale real-world validation, broader benchmark coverage, stronger memory-sharing safeguards, and richer multimodal small-model training remain necessary.Limitations include noisy perception, imperfect actuation, network latency, safety constraints, heterogeneous embodiments, limited scene and social-interaction diversity, reliance on structured traces and post-hoc feedback, and text-state-only semantic sandboxes; future work will report full benchmark evaluation and release EmbodiedWorldBench.Source: §6 · p. 28

Benchmark Construction

• The Benchmark Construction passage lists Jiayi Tian, Fei Wang, Tianlin Zhang, Yuting Xu, Xu Hu, Xiuxian Li, Wei Zhang, Jia Lu, Yiyan Ji, Jingbo Wang, and Ziteng Feng.No benchmark construction methods or specifications are provided in the passage.Source: Benchmark Construction · p. 29

Advisor

• The Advisor passage names Mu Xu and Wenbin Tang.Source: Advisor · p. 29
• The authors thank the broader team, particularly Wenbin Tang, Shihui Su, Zixiao Tang, Zhining Gu, Yang Cai, Linbo Zheng, and Jingjing Ma.Source: Advisor · p. 29

A Memory Benchmark Judge Prompts

• Three LLM-judged memory benchmarks use separate prompts matched to their answer formats and scoring protocols.The appendix identifies OpenEQA, Mem-Gallery, and LoCoMo as the benchmarks covered.Source: A Memory Benchmark Judge Prompts · p. 37, §A.3 · p. 38

A.1 OpenEQA Judge Prompt

• OpenEQA uses a 1–5 correctness score, accepting semantic equivalents while requiring matching polarity for object states and matching counts unless the reference is approximate.Judges must not reward details that contradict the reference and return JSON with a score, match flag, and brief reason.Source: §A.1 · p. 37

A.2 Mem-Gallery Judge Prompt

• Mem-Gallery permits scores of 0, 0.25, 0.5, 0.75, or 1, with an optional official-style parser mapping them to 0, 0.5, or 1.The strict parser preserves all five levels; the rubric ranges from incorrect or missing to correct or semantically equivalent, and the prompt requests JSON.Source: §A.2 · p. 37

A.3 LoCoMo Judge Prompt

• LoCoMo marks an answer CORRECT when it includes at least one correct gold-answer item, accepts paraphrases, and does not penalize extra detail.The prompt directs judges to assess meaning rather than exact wording.Source: §A.3 · p. 38
• LoCoMo accepts dates within 14 days, durations within 50%, and relative or vague dates consistent with the conversation context.Converting “last year” to the year relative to the conversation year also counts as correct.Source: §A.3 · p. 38
• LoCoMo accepts answers capturing the gold answer’s core idea, same-valence sentiment about the same event, or the same named referent despite descriptive differences.When conversation evidence is supplied, it can support acceptance even if the answer diverges from the gold answer; evidence is not used to reject answers more strictly.Source: §A.3 · p. 38
• LoCoMo marks an answer WRONG only when it contains no correct gold-answer items or addresses a completely different topic.The prompt requests JSON containing one-sentence reasoning and a CORRECT or WRONG label.Source: §A.3 · p. 38

B Lifelong Self-Evolution Process Example

• The loop evaluates the incumbent, diagnoses failure clusters, proposes generic writer, retriever, answerer, or frame-policy assets, applies safety and regression gates, then confirms accepted assets as a stack.Safety review rejects executable code, schema migration, direct answer resolvers, fixed answer tables, and qid/gold-answer rules; acceptance requires target improvement without global, low-score-count, or protected-category regression.Source: B Lifelong Self-Evolution Process… · pp. 38–39
• In split_00, the diagnosed failure was missing graph materialization of grounded object nodes or relations, leading retrieval to return unrelated records and the answerer to produce unsupported or missing answers.The resulting generic policy ranks observation-grounded object memories ahead of broad scene summaries when queries contain object, place, or spatial cues, while preserving source and frame evidence and avoiding ungrounded inference.Source: B Lifelong Self-Evolution Process… · pp. 39–40
• 0.6545 normalized stack mean versus 0.6053 baseline (+0.0492), with low-score count falling from 91 to 77; object-localization improved about +0.073 without object-state regression.These were outcomes of the split_00 stack containing the accepted retriever asset.Source: B Lifelong Self-Evolution Process… · pp. 39–40, Table 8 · p. 39
• Across the eight-split run, candidate-level wins alone were insufficient: only changes whose full evolved stack passed confirmation were retained, and most proposed writer policies were rejected for regression risks.The accepted split_00 retriever asset was logged provisionally; the successful change exposed existing graph evidence more reliably rather than adding answer-specific rules.Source: B Lifelong Self-Evolution Process… · p. 40, Table 9 · p. 40