PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents

Yehang Zhang, Jianchong Su, Haojian Huang, Yifan Chang, Tianhao Zhou, Xinli Xu, Yingjie Xu, Yinchuan Li, Zexi Li, Ying-Cong Chen

arXiv:2606.18847v1cs.AI

TL;DR

Long-horizon embodied agents need memory that tracks evolving household states, but existing benchmarks rarely test this under persistent, partially observable interaction. WorldLines benchmarks Memory QA and Embodied Task Planning with evidence-linked household traces and introduces ObsMem, while experiments show persistent challenges and stronger Memory QA performance with observer-grounded memory.

  • Problem

    Existing embodied benchmarks typically use short episodes without persistent state, leaving long-horizon memory use in dynamic, partially observable household environments insufficiently evaluated.

  • Method

    WorldLines builds temporally extended household traces for Memory QA and Embodied Task Planning, while ObsMem separates events, world states, and beliefs for state-aware reasoning.

  • Results

    Existing memory systems struggle with overwritten states, uncertainty, and embodied decisions, while ObsMem achieves the strongest overall Memory QA performance, improving Judge by 0.138 over A-mem.

  • Takeaways & Limitations

    The findings highlight the need for memory architectures designed for stateful embodied interaction.

  • Takeaways & Limitations

    WorldLines uses simulated household environments and does not fully cover perception noise, actuation errors, or open-ended human behavior in real homes.

Abstract

from arXiv · show

To assist humans over extended periods in real homes, embodied agents must remember user routines, world states, and past interactions. Existing long-term memory benchmarks mainly evaluate language-centric retrieval and question answering, while embodied benchmarks often focus on short-horizon task execution without testing long-term memory use in dynamic environments. We introduce WorldLines, a project-driven benchmark for long-horizon embodied household assistance. It constructs temporally extended household traces with dialogues, actions, execution feedback, object and device state changes, and converts them into evidence-linked samples for Memory QA and Embodied Task Planning. We further propose ObsMem, an observer-grounded memory framework that maintains visibility-aware memories and action-native state trails for state-aware decisions. Experiments reveal persistent challenges in partial observability, overwritten world states, and translating long-term memory into embodied plans, while ObsMem offers a stronger reference architecture for this setting.

1 Introduction

WorldLines frames long-horizon household assistance as persistent state maintenance under partial observability, linking routines, world changes, evidence, and executable actions across interactions. It introduces a benchmark and ObsMem framework that evaluate and support memory-grounded embodied planning in dynamic environments.

  • Motivation: Long-horizon embodied agents must maintain a stateful view of an evolving world, not merely remember past interactions.Service requests depend on user routines, object states, device settings, and recent events.
  • Problem: Existing memory and embodied benchmarks separately omit important links between retrieval, persistent physical states, action feedback, and executable constraints.Memory benchmarks decouple memory from physical transitions, while embodied benchmarks usually use short episodes where states do not persist across interactions.
  • WorldLines: WorldLines generates extended traces containing dialogue, activity, actions, device control, execution feedback, and world-state changes for evidence-linked Memory QA and Embodied Task Planning.The benchmark covers dynamic, partially observable environments and evaluates long-horizon stateful embodied agents.
  • ObsMem: ObsMem separates historical evidence, structured world states, and agent beliefs to maintain persistent state and support embodied decisions under partial observability.Its records include event evidence, state trails, and belief records for long-horizon embodied QA and planning.

2 Related Work

Prior long-term memory benchmarks largely evaluate conversational or multimodal retrieval, question answering, updating, consistency, and project-oriented interaction, while embodied benchmarks emphasize household tasks, navigation, interaction, and spatial reasoning. Existing memory architectures and embodied-agent methods provide external storage, structured world knowledge, observation retrieval, and reusable skills, but long-term memory is not explicitly evaluated in embodied task completion.

  • Long-Term Memory Benchmarks: Long-term memory benchmarks such as LoCoMo, LongMemEval, HaluMem, and RealMem establish protocols for dialogue memory, memory updating, consistency, hallucination, and project-oriented interaction.These benchmarks remain largely text-centric.
  • Embodied Benchmarks: Embodied benchmarks cover language-conditioned household tasks, simulated navigation and interaction, and dynamic spatial reasoning under partial observability.ALFWorld, ProcTHOR, Habitat 3.0, and PARTNR represent these complementary settings, with PARTNR centered on game-like navigation and elimination tasks.
  • Research Gap: Existing embodied benchmarks do not explicitly evaluate long-term memory in embodied task completion.This limitation is stated for benchmarks centered on game-like navigation and elimination tasks under partial observability.
  • Memory Architectures: Existing LLM-based agents use external memory to store, update, and retrieve information beyond a single context window.MemGPT uses hierarchical memory tiers, MemoryBank accumulates user-specific conversational memories, and later systems improve memory management through operating-system-inspired scheduling and scalable extraction and update pipelines.
  • Embodied Memory Methods: Embodied agents maintain structured object, spatial, and relational knowledge, retrieve past observations, or store reusable skills and programs for future tasks.MEMENTO studies personalized embodied assistance, while semantic maps, scene graphs, observation retrieval, and skill storage support embodied decision making and planning.

3 WorldLines Benchmark Construction

WorldLines constructs long-horizon household traces by grounding executable worlds, generating persistent multi-day projects, simulating validated state-changing interactions, and carrying memory across days. It converts these traces into cutoff-controlled samples for Memory QA and Embodied Task Planning, evaluating state maintenance and state-consistent decisions from pre-cutoff history.

  • World grounding: World grounding defines household entities and executable interactions through Habitat/HSSD scenes, curated objects, receptacles, devices, and separate semantic and executable scene views.The executable view supports validation of object instances, device states, and openable components.
  • Project planning: Project planning creates persistent multi-day household threads with activities, participants, spaces, entities, temporal preferences, and constraints.Projects include routine support, meal preparation, home organization, and device coordination.
  • Closed-loop trace simulation: Closed-loop simulation generates dialogue, human activity, robot actions, and device operations, while a deterministic executor records only outcomes satisfying entity, affordance, and precondition checks.Each day conditions generation on the grounded world, active projects, accumulated state, and carry-forward notes.
  • Carry-forward memory: Carry-forward memory extracts changed states, preferences, unresolved plans, recent events, and conflicts so later days depend on earlier household history.Notes from events, execution feedback, and state changes are passed forward with accumulated state.
  • Evaluation samples: Evaluation samples use programmatically determined cutoffs, ground truths, and evidence chains, testing Memory QA and Embodied Task Planning from pre-cutoff history without future-information leakage.Memory QA covers states, overwritten states, temporal reasoning, preferences, routines, and source-aware questions; planning requires executable, state-consistent decisions.

4 ObsMem: Observer-Grounded Memory

ObsMem treats long-horizon embodied memory as an online, observer-grounded process that preserves typed evidence rather than compressing all history into undifferentiated text. It maintains distinct views for events, states, beliefs, and commitments, then composes task-specific evidence for grounded answering and state-aware planning.

  • Memory writing: ObsMem gates events by observer provenance, distinguishing directly observed evidence from reported claims before writing them into memory.An event enters the robot’s memory only when the robot belongs to the event’s observer set; non-robot utterances additionally create reported atoms.
  • Memory writing: Visible events enter the Event Track, executable actions produce structured State Track facts, and requests or schedules enter the Commitment Track.This preserves each record’s semantic role at write time instead of requiring retrieval to infer it from plain text.
  • Memory updating: ObsMem applies role-specific updates: events remain append-only, states retain current snapshots plus historical trails, and beliefs track epistemic reliability under partial observability.Historical state facts are preserved when newer facts overwrite the current snapshot, enabling both current-state and overwritten-state queries.
  • Memory updating: Beliefs become stale, uncertain, or contradicted when reports, intervening events, or conflicting evidence reduce confidence in the latest state.The framework uses deterministic rules with fixed implementation hyperparameters to update epistemic status.
  • Memory retrieval: At query time, ObsMem composes memory views according to task intent instead of performing one undifferentiated similarity search.Current-state questions use State and Belief, past-event questions use Event and Episode, commitment questions use Commitment and Event, and planning combines state, history, obligations, and uncertainty.

5 Experiment

WorldLines evaluates memory systems on evidence-linked Memory QA and embodied planning under delayed, cross-day state changes. ObsMem achieves the strongest reported results, while ablations and planning analyses show the importance of evidence selection, typed state trails, and state-aware execution.

  • Evaluation Setup: WorldLines evaluates Mem0, A-mem, MemoryOS, GraphMem, and ObsMem on Memory QA and Embodied Task Planning using the same cutoff-controlled visible history.For QA, systems pass at most five retrieved records to the answer generator, with retrieval metrics computed over those top-five records.
  • Memory QA Results: ObsMem achieves the strongest overall Memory QA performance, improving Judge by 0.138 over A-mem and Event R@5 by 0.159 over Mem0.Its typed state trails and event-grounded retrieval help recover concrete household evidence needed for answering.
  • Memory QA Results: High Session Any@5 does not ensure embodied memory quality because exact state-changing event recovery remains substantially harder, especially for mutable-state questions.ObsMem performs best on StateMultiHop and StateSingleHop questions, while flat-text memory can retrieve salient routines but miss anomalous state updates.
  • Ablations: Removing evidence selection drops Judge from 0.699 to 0.435, while removing episode consolidation reduces Judge to 0.554 and Event R@5 to 0.419.These ablations show that selecting and combining evidence across memory views, synthesizing episodes, and resolving conflicts are central to answer quality.
  • Embodied Task Planning: ObsMem obtains the highest Plan Judge score and performs especially well on state consistency, precondition validity, and memory use in executable household planning.Planning requires checking object locations, container states, device states, and action preconditions before producing action sequences.

6 Conclusion

WorldLines benchmarks long-horizon stateful embodied agents in dynamic, partially observable households by testing persistent world-state maintenance across interactions and its use in Memory QA and Embodied Task Planning. ObsMem separates event evidence, structured world states, and agent beliefs to support state-aware reasoning, while experiments expose persistent difficulties with overwritten states, uncertainty, and embodied decision-making.

  • WorldLines: WorldLines evaluates long-horizon stateful embodied agents in dynamic, partially observable household environments.It focuses on maintaining persistent world states across dialogue, human activity, robot actions, device changes, and execution feedback.
  • WorldLines: The benchmark tests whether persistent world states support Memory QA and Embodied Task Planning.Its traces span dialogue, human activity, robot actions, device changes, and execution feedback.
  • ObsMem: ObsMem separates event evidence, structured world states, and agent beliefs for state-aware reasoning under partial observability.The framework is observer-grounded and designed to support reasoning when the environment is only partially observable.
  • Limitations: Existing memory systems struggle with overwritten states, uncertainty, and translating long-term memory into embodied decisions.These findings motivate memory architectures designed specifically for stateful embodied interaction.

7 Limitations … A.2 QA Verification Protocol

WorldLines provides controlled, evidence-annotated evaluation but does not fully represent real-home conditions. The study also specifies standardized baseline evaluation and manually verifies QA items against cutoff-valid supporting evidence.

  • 7 Limitations: WorldLines uses simulated household environments that omit perception noise, actuation errors, and open-ended human behavior in real homes.The controlled setting enables precise annotation of evidence chains, cutoffs, state changes, and executable constraints.
  • 7 Limitations: Future work can extend WorldLines to real robot logs, visual observations, and full physical simulation.
  • 7 Limitations: ObsMem requires structured traces with entity identifiers, visibility annotations, and action schemas that real deployment would obtain from perception, localization, and grounding modules.
  • 7 Limitations: ObsMem adds latency through typed retrieval and belief-aware evidence selection, motivating more efficient retrieval and integration with visual perception and execution feedback.
  • A.1 Baseline Implementation Details: Mem0, A-mem, MemoryOS, and GraphMem serve as representative long-term memory baselines using official implementations when available and retaining recommended memory-update procedures.
  • A.1 Baseline Implementation Details: All methods receive the same cutoff-controlled visible household history before each query, with a shared answer-generation context budget.
  • A.1 Baseline Implementation Details: Only the top five final retrieved records reach answer generation and retrieval metrics, while ObsMem may return fewer than five when high-confidence typed evidence is scarce.
  • A.2 QA Verification Protocol: Annotators manually verify that each QA item is answerable before cutoff, entailed by supporting evidence, and independent of post-cutoff information or hidden evaluator state.

A.3 Judge Validation Protocol · A.4 Radar Visualization by Embodied Memory Type

The paper validates GPT-4o as an answer-correctness judge against human annotations and visualizes how embodied memory systems perform across memory types. The results support judge scores as a reasonable proxy while showing that broad session retrieval may miss exact state-changing events.

  • A.3 Judge Validation Protocol: The judge protocol is validated on 80 sampled system outputs covering all evaluated methods and question families.Five human annotators independently scored each answer using the shared 0–3 correctness rubric.
  • A.3 Judge Validation Protocol: Automatic and human evaluation use the same 0–3 rubric, with answers categorized as incorrect, partially correct, or correct.GPT-4o decisions are compared against the majority human label.
  • A.3 Judge Validation Protocol: Fleiss’ κ = 0.71 indicates substantial human agreement, while GPT-4o reaches 87.5% agreement with majority human labels and 0.82 Spearman correlation with averaged human scores.These results support using GPT-4o judge scores as a reasonable proxy for answer correctness.
  • A.4 Radar Visualization by Embodied Memory Type: Figure 8 visualizes QA score, session recall, and event recall across representative embodied memory categories for each memory system.ObsMem shows a more balanced profile across state-centric categories, whereas generic systems often retrieve coarse sessions without equivalent event grounding.
  • A.3 Judge Validation Protocol: Session Any@5 and Event R@5 provide judge-independent diagnostics for evidence grounding.The metrics complement judge scores by assessing retrieval-based grounding directly.
  • A.4 Radar Visualization by Embodied Memory Type: The contrast between Session Any@5 and Event R@5 shows that broad session retrieval can fail to recover the exact state-changing events required by an answer.Methods may retrieve broadly relevant sessions without identifying the decisive events.

A.5 Ablation Variant Details · A.6 Downstream Embodied Planning Probe · A.7 Full Efficiency and Context-Cost Statistics

The appendix isolates ObsMem mechanisms, probes whether its memories support executable embodied planning, and reports efficiency and context-cost considerations under a five-record budget. Results indicate promising state-aware planning gains, while typed retrieval can reduce passed records but structured retrieval increases latency.

  • A.5 Ablation Variant Details: The ablation study removes one ObsMem mechanism at a time while keeping ingestion, retrieval, answer generation, judging, samples, backbone, and prompts otherwise fixed.It uses a 62-sample diagnostic QA subset spanning StateMultiHop, StateSingleHop, and TemporalMemory questions.
  • A.5 Ablation Variant Details: The ablation variants isolate individual mechanisms in the ObsMem memory pipeline without changing benchmark samples, backbone model, judge, or answer-generation prompt format.
  • A.6 Downstream Embodied Planning Probe: The planning probe evaluates state consistency, precondition validity, memory use, and action order, with Plan Judge averaging these dimensions.
  • A.6 Downstream Embodied Planning Probe: 7.6 target actions and 3.1 remembered state constraints are required on average per planning instance in the action-dense downstream probe.Constraints include object locations, container states, device settings, and action preconditions.
  • A.6 Downstream Embodied Planning Probe: ObsMem shows promising gains across all planning dimensions, especially state consistency and precondition validity.The probe is smaller-sample and tests executable, state-aware decisions using remembered household states.
  • A.7 Full Efficiency and Context-Cost Statistics: All systems pass at most five retrieved records to answer generation, while ObsMem often passes fewer because typed retrieval abstains from weakly matched evidence.
  • A.7 Full Efficiency and Context-Cost Statistics: GraphMem can create larger prompt contexts because graph-expanded records include neighboring node descriptions despite the five-record cap.
  • A.7 Full Efficiency and Context-Cost Statistics: Avg. Passed Records measures final context size rather than retrieval correctness, while ObsMem incurs higher latency from structured state and belief-aware retrieval.

A.8 Additional Metrics · B Replay and Edited Scene Examples

The section adds event-level diagnostics to expose the gap between coarse session retrieval and precise event-level grounding, then illustrates replay in edited Habitat/HSSD household scenes. Evaluation relies on structured event logs and state-linked annotations rather than visualizations alone.

  • A.8 Additional Metrics: Table 11 reports additional question-family event-level metrics.These diagnostics complement the compact main-table results.
  • A.8 Additional Metrics: The additional diagnostics expose differences between coarse session retrieval and precise event-level grounding.They are intended to show a gap that compact main-table results may not capture.
  • B Replay and Edited Scene Examples: WorldLines uses Habitat/HSSD household scenes rather than text-only interaction logs.This establishes the benchmark’s scene-based embodied setting.
  • B Replay and Edited Scene Examples: Base scenes are manually curated with interaction-relevant objects, valid receptacles, controllable devices, and household roles.The curation defines the elements needed for each question family.
  • B Replay and Edited Scene Examples: Generated event traces can be replayed against the edited scene state.Replay supports qualitative inspection and illustration through the scene visualizations.

C Benchmark Taxonomy and Action Space · D Prompt Templates

WorldLines organizes trace generation around project types, memory targets, and executable skills, then uses staged prompts to move from abstract household themes to evaluated, evidence-linked traces. Its closed-loop protocol separates planning, acting, execution, and memory evaluation so generated behavior remains grounded in evolving scene state.

  • C Benchmark Taxonomy and Action Space: WorldLines’ controlled vocabulary comprises project types, memory targets, and executable skills for generating long-horizon household traces.Project types organize household themes, memory targets define intended challenges, and the action space supplies executable skills.
  • C Benchmark Taxonomy and Action Space: Project types cover recurring household routines, habit support, organization, health and lifestyle, meals, work or study, device coordination, and family comfort.These themes include household_routine, routine_support, household_organization, health_lifestyle, meal_preparation, work_study_support, digital_device_coordination, and family_comfort.
  • C Benchmark Taxonomy and Action Space: Memory targets span object locations, temporal and device states, preferences, routines, planning dependencies, hidden states, and social context.The taxonomy includes object_location, temporal_state, device_state, preference, routine, planning_dependency, hidden_state, and social_context.
  • C Benchmark Taxonomy and Action Space: The executable action space includes navigation, inspection, picking, placing, opening, closing, device-state modification, and actor handoffs.Actions are constrained by scene affordances and interaction conditions such as co-location, empty hands, and valid destinations.
  • D Prompt Templates: Prompt templates are filled programmatically and stage generation from semantic project candidates through multi-day beats and ordered session intents.Candidate projects avoid concrete object IDs and actions; beat planning avoids concrete sessions and state changes; session planning adds situations, timing, visibility, and narrative pressure.
  • D.4 Closed-Loop Agentic Trace Generation: The final trace generator uses a closed-loop director–actor–executor protocol: the director sets up sessions, actors propose executable events, and the executor applies state changes and derives visibility.Actor proposals are validated before becoming trace events, while the director controls pacing, continuity, and temporal conflicts without writing executable actions or state_diffs.
  • D.5 Evaluation and ObsMem Runtime Prompts: Evaluation prompts generate 0–3 evidence-linked Memory QA candidates, judge answers against event-history information, and convert questions into typed retrieval views for ObsMem evidence selection.The Session Examiner proposes future recall questions, the LLM-as-Judge scores information use, and the ObsMem query planner produces a compact JSON retrieval plan without answering.
Loading 2606.18847v1…