PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Tianyidan Xie, Shenyi Wang, Qiang Tang, Mingjie Wang, Zhicheng Qiu, Xuanfu Li, Zhan Xu, Jian Yang, Lanjun Wang, Zili Yi

arXiv:2609.04802v1cs.CVcs.AI

TL;DR

Long-horizon embodied agents lack language-queryable per-object timelines for dynamic state transitions. The paper introduces LTE, a hybrid representation combining motion captions with sparse spatial and visual anchors, and evaluates it with SMB. LTE-based retrieval outperforms prior baselines on SMB while also compressing trajectories and improving established Ego4D results.

  • Problem

    Existing memory systems either discard fine-grained motion, retain coordinates without semantic state predicates, or organize records around immediate tasks rather than per-object timelines.

  • Method

    LTE encodes each object’s motion using language-described phases, sparse 3D anchors, and visual crops, with adaptive handling of tracking gaps and trajectory complexity.

  • Results

    45.3% STR and 48.7% LOR success on SMB exceed the strongest VLM baseline, while Ego4D NLQ reaches 28.75%/55.10% R@1/R@5 and LTE compresses trajectories 8.7×–26.1×.

  • Takeaways & Limitations

    The results support language at the level of motion phases as a useful middle layer between geometric spatial memory and clip-level video-language representations.

  • Takeaways & Limitations

    The evaluation uses atomic object tracking and long-horizon stress tests in EgoLife home environments, leaving part-level dynamics and broader environments for future work.

Abstract

from arXiv · show

Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$\,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.

1 Introduction

Long-horizon embodied agents need per-object timelines whose semantic state transitions, locations, and moments remain jointly queryable. LTE addresses this gap with a hybrid trajectory representation, while SMB evaluates retrieval across multi-day recordings.

  • Motivation: A single query may require identifying an object, matching a semantic state change, and recovering its location at a specific time.The motivating example asks where a washed apple was last placed after six hours of observation.
  • Gap: Existing systems either drop fine-grained motion, retain coordinates without semantic predicates, or summarize objects only when task-relevant.Consequently, they do not provide per-object timelines whose state transitions are queryable in language.
  • LTE: LTE records each object’s motion with captions, sparse 3D anchors, and visual crops, matching semantic, spatial, and identity queries.Adaptive compression uses last-seen anchors for tracking gaps, few waypoints for simple motion, and more for complex motion.
  • Evaluation: 45.3% STR and 48.7% LOR success on SMB exceed the strongest VLM baseline’s 31.9% and 34.4%, respectively.SMB contains 600 queries over multi-day EgoLife recordings, with sessions reaching 50 h.
  • Results: +13.4 points on STR and +14.3 points on LOR over the strongest VLM baseline accompany first-place zero-shot performance on established Ego4D tasks.The integrated system also achieves compression factors of 8.7× to 26.1× with sub-second query latency on 24 h video.

2 Related Work

Prior spatial, trajectory, and video-memory systems provide pieces of long-horizon retrieval but do not offer language-indexed per-object timelines of state changes. LTE combines spatial indexing with linguistic trajectories to address this gap.

  • Spatial memory: Spatial memory systems preserve geometry or semantic object attributes but generally represent semantics as static snapshots rather than temporal state sequences.Keyframe-based memories also subsample observations without providing language-grounded motion phases.
  • Trajectory representation: Geometric and semantic trajectory methods compress coordinates or labels, while video agents organize observations into events without language-grounded per-object motion abstractions.These representations do not provide the specific hybrid timeline targeted by LTE.
  • Long-horizon video: Ego4D offers short natural-language-query videos, whereas EgoLife supplies continuous multi-day recordings for long-horizon evaluation.Video-language models lack explicit spatial structures or trajectory abstractions and lose global context across sequential clips.
  • LTE: LTE addresses these gaps through spatial indexing combined with per-object linguistic trajectories.This positions language-described motion phases between purely geometric and clip-level video-language representations.

3 Method

The method converts continuous egocentric video into structured, indexed memory that supports spatial, temporal, semantic, and visual retrieval. LTE compresses each dynamic object’s trajectory through language, adaptive spatial anchors, and visual evidence, while complementary views route queries efficiently.

  • Problem setup: The system models an hours-to-days video stream as a queryable memory supporting spatial, temporal, and natural-language semantic queries.The architecture separates perception, memory organization, and query processing.
  • Architecture: Perception converts video into structured observations, while the memory architecture records LTE objects and uses an octree for spatial pruning.Five complementary views feed a router serving NLQ, VQ2D, STR, and LOR.
  • LTE representation: LTE represents motion phases with captions, critical positions with spatial anchors, and identity evidence with visual crops.The hybrid design preserves semantic accessibility while retaining geometric precision and visual verification.
  • LTE construction: Tracking-gap intervals are anchored at the last observed position, while motion intervals receive captions and geometry-dependent Douglas-Peucker waypoints.The VLM generates motion captions for each interval, and spatial anchors are placed adaptively.
  • Query processing: Semantic, spatial, visual, and temporal queries use captions, anchor positions, visual embeddings, and interpolation between adjacent anchors.The channels are linked so matched evidence resolves to object identity, time, and spatial context.
  • Query processing: All four tasks share a parse–route–filter–aggregate pipeline, with confidence labels distinguishing observed, extrapolated, and historical-only evidence.Constraint-dependent processing avoids exhaustive frame scans as video duration grows.

4 Experiments

The experiments evaluate LTE across Ego4D and the new SMB benchmarks, showing stronger long-horizon retrieval, efficient querying, and complementary contributions from its memory components, while tracking remains a major failure source.

  • Experimental setup: SMB evaluates semantic trajectory retrieval and long-horizon object retrieval with 300 queries each over multi-day EgoLife recordings.STR uses motion state, trajectory, temporal windows, and semantic spatial regions; LOR retrieves last occurrences over 2–24-hour lookbacks.
  • Ego4D results: 28.75%/55.10% R@1/R@5 at IoU=0.3 improves over EgoVLPv2’s 12.95%/23.80% and approaches OSGNet’s 32.56%/59.82%.These results are reported for Ego4D NLQ temporal localization.
  • Ego4D results: 0.36 stAP and 59.5% success provide the strongest zero-shot VQ2D performance, slightly exceeding RELOCATE’s 0.33 stAP and 58.0% success.The result is attributed to object-centric visual anchors followed by temporal filtering.
  • SMB results: 45.3% STR success and 48.7% LOR success outperform the strongest VLM baseline by +13.4 STR and +14.3 LOR.The system also exceeds KFMem on STR, where KFMem reaches 19.8% because it cannot represent motion histories.
  • Failure analysis: Tracking errors dominate qualitative failures, with ID switches alone accounting for 34% of STR errors.Figure 3 attributes most failures to upstream processing rather than the memory representation.
  • Efficiency: At 24 h, query latency remains 0.43 s and total time for 300 SMB queries is 2.2 minutes, versus 98.3 s per query and 8.2 hours for the VLM baseline.Memory grows from 45 to 134 MB for 12× more video, reflecting sub-linear growth.
  • Ablations: Removing text captions reduces STR by −11.8, removing visual anchors reduces VQ2D success by −6.1, and removing the octree reduces LOR by −3.5.The full three-channel system delivers +16.5 STR, compared with +11.4 from the individual channel gains summed independently.

5 Discussion

LTE represents dynamic objects as per-object linguistic timelines, while SMB evaluates long-horizon spatial memory. The paper reports channel complementarity and identifies articulated dynamics and broader environments as future extensions.

  • 5 Discussion: LTE encodes each object’s motion as language-described phases anchored to sparse 3D positions and visual crops.
  • 5 Discussion: Text captions are the dominant single LTE channel, with removal reducing performance by 11.8 points versus 16.5 points for removing all channels.
  • 5 Discussion: SMB is built from EgoLife multi-day recordings and evaluates semantic trajectory retrieval and long-horizon object retrieval.
  • 5 Discussion: The current scope tracks objects atomically and evaluates long-horizon stress tests in EgoLife home environments.
  • 5 Discussion: Future expansion includes articulated part dynamics and evaluation across office, warehouse, and outdoor recordings.
  • 5 Discussion: The benchmark construction uses tracked trajectories, manual verification, object-centric state annotations, spatial regions, and semantic temporal descriptions.

A.4 Long-Horizon Object Retrieval Task

Long-Horizon Object Retrieval finds the last occurrence of described objects within temporal lookback windows. Its annotation procedure searches chronologically ordered video segments in reverse time and records the final occurrence frame and box.

  • A.4 Long-Horizon Object Retrieval Task: The task retrieves the last occurrence of described objects within 2 h, 6 h, 12 h, and 24 h lookback windows.
  • A.4 Long-Horizon Object Retrieval Task: Annotators inspect chronologically sorted segments from newest to oldest to identify the last occurrence accurately.
  • A.4 Long-Horizon Object Retrieval Task: 40.0% of queries include spatial hints, 45.3% include temporal hints, and 22.7% include both.
  • A.4 Long-Horizon Object Retrieval Task: Ground truth records the last-occurrence frame t* and corresponding bounding box b*, with typical spans of 3–5 seconds.
  • A.4 Long-Horizon Object Retrieval Task: Inter-annotator agreement is 0.78 for timestamp localization within 3 seconds and 0.72 for bounding boxes at IoU ≥0.5.

B.2 Model Specifications

The system combines segmentation, reconstruction, language and visual models, spatial indexing, trajectory encoding, and adaptive scene understanding for long-horizon retrieval.

  • B.2 Model Specifications: The pipeline uses SAM3 for tracking, ViPE for 3D reconstruction, Qwen models for parsing and captioning, and DINOv2, SigLIP2, and Whisper for multimodal processing.
  • B.2 Model Specifications: The octree uses maximum depth dmax = 8, minimum node size 0.1m × 0.1m × 0.1m, and a 20m × 20m × 5m root.
  • B.2 Model Specifications: LTE uses Douglas–Peucker simplification, bounded anchor spacing, a 2-second tracking-gap threshold, and captions at motion-phase transitions.
  • B.2 Model Specifications: Adaptive scene understanding processes 3-second windows at 3 FPS to infer room labels and select salient keyframes.
  • B.2 Model Specifications: Algorithm 1 increases the sampling stride during stable room labels and backtracks when labels change to locate boundaries.
  • B.2 Model Specifications: Query processing uses cosine-similarity thresholds of 0.7 for VQ2D, 0.8 for STR, and 0.75 for object-semantic fallback.

C End-to-End Timing

End-to-end timing separates one-time offline memory construction from subsequent query processing. The reported break-even point indicates when repeated queries offset construction overhead.

  • C End-to-End Timing: Table 8 reports the complete pipeline timing from raw video to query response on a single A800 GPU.
  • C End-to-End Timing: The memory construction cost is paid once, after which persistent environments can use sub-second query latency.
  • C End-to-End Timing: Approximately 1,545 queries on 24 h of video mark the break-even point relative to Q3VL-235B+GD.
  • C End-to-End Timing: The paper identifies adaptive frame skipping, incremental construction, and lighter perception models as ways to reduce construction time.

D.1 Long-Horizon Object Retrieval Breakdown

The LTE system remains effective across longer horizons, query types, and stricter localization criteria, while its full representation benefits from complementary channels.

  • Long-Horizon Object Retrieval Breakdown: 4.4-point degradation from 2 h to 24 h (51.2% → 46.8%) is lower than Qwen3-VL-235B’s 9.8-point degradation (39.1% → 29.3%).The analysis attributes this stability to octree spatial indexing and object-centric tracking rather than sequential clip processing.
  • Long-Horizon Object Retrieval Breakdown: The performance advantage holds at IoU ≥0.5, with gains of +12.4 STR and +13.5 LOR over the comparison system.This stricter-threshold result indicates that the reported advantage is not limited to lenient IoU evaluation.
  • Long-Horizon Object Retrieval Breakdown: 49.5% is achieved for queries containing both spatial and temporal hints, while two-object queries perform worse because tracking errors can compound across entities.Combined constraints enable more precise filtering, whereas multiple tracked entities increase the opportunity for identity errors.

E Error Analysis

The error analysis identifies tracking, localization, and caption ambiguity as primary failure sources, while describing mechanisms for constructing linked object, text, event, and spatial records.

  • Failure Analysis: Tracking failures are the dominant error source, accounting for 34% of STR failures and 28% of LOR failures.They include identity switches, lost tracks after more than 2 seconds out of view, and small-object detection or identification errors at scales such as ≤30 × 30 pixels.
  • Failure Analysis: Caption ambiguity causes 28% of STR failures because descriptions may omit source-location information needed to match motion-path queries.A caption such as “object was placed on surface” can lack the specificity required for precise state matching.
  • Failure Analysis: Extended ViPE operation produces point-cloud drift responsible for 18% of STR errors and 24% of LOR errors.Douglas–Peucker simplification absorbs frame-level depth noise, while room labels provide a drift-robust spatial fallback.
  • Observation Limits: A single ego-camera cannot observe off-frame state changes, so the system reports only the last known state when an object moves out of view.Confidence labels distinguish directly observed objects, extrapolated last-seen positions, and objects supported only by historical LTE data.
  • Memory Construction: The pipeline links motion captions, room labels, and speech transcripts to object, spatial, temporal, and event records for retrieval.LTE motion events link participating objects, while Text view stores embedded captions, room labels, and timestamped transcripts in a shared vector index.

G.4 Baseline Implementation Details

The baseline implementations compare LTE with clip-based VLM processing, keyframe memory, and event-based video memory using standardized query and extraction procedures.

  • Keyframe Memory: Keyframe Memory selects approximately 600 diverse keyframes per 24 h, back-projects detected objects into 3D, and retrieves candidates using description and spatial-temporal constraints.The implementation follows the 3D-Mem paradigm and uses the same ViPE reconstruction as the LTE system.
  • VideoAgent: VideoAgent uses an event-based memory across processed clips and an LLM agent for querying, with Qwen3-VL-8B as its underlying VLM.The comparison adopts the public implementation and default configuration for the SMB evaluation.
  • Scene Understanding: Room labeling classifies frames into room categories and selects room-entry, room-exit, and salient in-room-event keyframes.The classifier also receives the previous window classification and marks whether a room transition was detected.
  • Motion Captioning: Object-motion prompts ask the VLM to describe direction, path, interactions, state transitions, and spatial context in one concise sentence.The highlighted object is identified by a bounding box, and the prompt emphasizes motion phases such as being picked up or placed down.
  • Speech Event Extraction: Speech-event extraction records activities, participants, objects, locations, temporal markers, and confidence from timestamped transcripts.Only events with at least medium confidence are retained, with object, spatial, temporal, and semantic fields normalized to controlled vocabularies.

I Limitations and Societal Impact

The system’s scope is limited by upstream tracking and reconstruction reliability, atomic object modeling, single-environment assumptions, and privacy requirements for shared-space deployment.

  • System Limitations: Tracking errors, point-cloud drift, and boundary-only captions can corrupt trajectory anchors, event links, spatial localization, and fine-grained state representation.The limitations include ID switches, missed detections, track fragmentation, multi-hour ViPE drift, and missed continuous-manipulation changes.
  • System Limitations: LTE models objects as atomic entities and does not explicitly represent articulated part dynamics such as door angles or drawer extension.Extending the representation to articulated parts is identified as a future direction.
  • System Limitations: The current system assumes a single persistent environment and does not address cross-environment re-identification or map alignment.Multi-environment operation would require robust object re-identification and alignment across maps.
  • Societal Impact: Long-horizon spatial memory stores detailed activity records, creating privacy considerations for deployment in shared spaces.The paper focuses on first-person, user-owned scenarios and identifies explicit consent and user-controlled retention as requirements for shared environments.
Loading 2609.04802v1…