PaperlayerView exact revision on arXiv

Source-linked AI summary

Describe Anything Anywhere At Any Moment

Nicolas Gorlo, Lukas Schmid, Luca Carlone

arXiv:2512.00565v1cs.CVcs.AIcs.RO

TL;DR

Large-scale, long-horizon perception needs spatial memory that combines geometric grounding, semantic detail, and real-time construction. DAAAM builds hierarchical 4D scene graphs using optimized observation selection and batched descriptions. It reports state-of-the-art results in spatio-temporal question answering and sequential task grounding, while noting throughput limits for some applications.

  • Problem

    Large-scale, long-horizon applications need memory that supports spatial reasoning and task planning while being built in real time, and existing approaches often excel in only one of geometric precision, semantic richness, or efficiency.

    Source: §1 · p. 2, §2 · p. 2

  • Method

    DAAAM uses optimized frame selection and batch annotation to combine detailed semantic descriptions with geometric mapping in a hierarchical 4D scene graph.

    Source: §1 · p. 2, §3 · pp. 3–4

  • Results

    DAAAM reports state-of-the-art results in both spatio-temporal question answering and sequential task grounding.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The 4D scene graph serves as spatio-temporal metric-semantic memory for embodied agents and can be queried by a tool-calling agent.

    Source: §1 · p. 2, §3 · p. 3

    A single worker annotates an average of 5.2 new fragments per second, which may be too slow for a dynamic aerial robot or VR headset.

    Source: §5 · p. 8

Abstract

from arXiv · show

Computer vision and robotics applications ranging from augmented reality to robot autonomy in large-scale environments require spatio-temporal memory frameworks that capture both geometric structure for accurate language-grounding as well as semantic detail. Existing methods face a tradeoff, where producing rich open-vocabulary descriptions comes at the expense of real-time performance when these descriptions have to be grounded in 3D. To address these challenges, we propose Describe Anything, Anywhere, at Any Moment (DAAAM), a novel spatio-temporal memory framework for large-scale and real-time 4D scene understanding. DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models, such as the Describe Anything Model (DAM), leveraging batch processing to speed up inference by an order of magnitude for online processing. It leverages such semantic understanding to build a hierarchical 4D scene graph (SG), which acts as an effective globally spatially and temporally consistent memory representation. DAAAM constructs 4D SGs with detailed, geometrically grounded descriptions while maintaining real-time performance. We show that DAAAM's 4D SG interfaces well with a tool-calling agent for inference and reasoning. We thoroughly evaluate DAAAM in the complex task of spatio-temporal question answering on the NaVQA benchmark and show its generalization capabilities for sequential task grounding on the SG3D benchmark. We further curate an extended OC-NaVQA benchmark for large-scale and long-time evaluations. DAAAM achieves state-of-the-art results in both tasks, improving OC-NaVQA question accuracy by 53.6%, position errors by 21.9%, temporal errors by 21.6%, and SG3D task grounding accuracy by 27.8% over the most competitive baselines, respectively. We release our data and code open-source.

1. Introduction

Large-scale, long-horizon understanding requires memory that supports spatial reasoning and task planning while operating in real time. DAAAM combines detailed descriptions with geometrically grounded, temporally consistent 4D scene graphs, using optimized frame selection and batch annotation to support online inference.

1. Introduction

  • Spatio-temporal memory must support spatial reasoning and task planning over long horizons and large environments while being built in real time.These requirements matter for applications such as robotics and augmented reality.Source: §1 · p. 2
  • Existing metric-semantic maps trade semantic detail against real-time computation, while frame-based multimodal annotations can lack 3D grounding and consistency across observations.Frame-indexed memories can therefore struggle with long-range spatial relationships and object quantities.Source: §1 · p. 2
  • DAAAM builds a hierarchical 4D scene graph by combining real-time metric-semantic mapping with detailed natural-language descriptions for observed entities.Its optimization-based frontend selects key observations for detailed descriptions, and batch processing speeds inference by an order of magnitude for online deployment.Source: §1 · p. 2
  • DAAAM reports state-of-the-art results in spatio-temporal question answering and sequential task grounding, and curates an extended question-answering benchmark.The framework also uses a tool-calling agent to query the 4D scene graph.Source: §1 · p. 2

2. Related Works

Spatial-memory research spans metric-semantic maps and view-based annotations, each offering different balances of geometric structure, semantic detail, and efficiency. DAAAM contributes a detailed, efficient spatial memory intended for embodied-agent reasoning and task grounding.

2. Related Works

  • Metric-semantic representations include point clouds, Gaussian splats, radiance fields, and TSDF volumes, but can have limited expressiveness, large memory footprints, and inefficient queries.Real-time scene graphs add hierarchy, yet often rely on closed-vocabulary semantics that constrain complex memory queries.Source: §2 · p. 2
  • Per-object multimodal-model queries produce detailed descriptions but are expensive, while frame-based annotations offer rich views without sufficient 3D grounding or cross-frame consistency.These view-based methods can struggle with precise spatial reasoning, long-range relationships, and object quantities.Source: §2 · p. 3
  • View-based memory systems store annotated frames or video segments for natural-language retrieval, while other approaches retrieve observed frames or represent observations as textual landmarks or tokens.Hybrid approaches seek to combine view-based flexibility with 3D structure in indoor environments.Source: §2 · p. 3
  • VLMs and LLMs gain spatial reasoning capabilities through scene-graph training, 3D-aware architectures, or spatially annotated video, but large-scale dynamic 3D environments remain challenging.Related systems also address temporal changes, task grounding, and functional relationships in scene graphs.Source: §2 · pp. 2–3
  • DAAAM positions its contribution as an efficient, detailed spatial memory for embodied agents performing complex reasoning and task grounding.The stated application context includes scene-graph planning, robotic affordances, structured LLM interfaces, embodied question answering, and navigation reasoning.Source: §2 · p. 3

3. Approach

DAAAM constructs a real-time 4D scene graph from RGB-D images and poses, combining tracked geometric fragments with detailed open-vocabulary annotations. Its pipeline selects and batches observations for annotation, reconciles the resulting memory over time, and organizes it into regions for retrieval and reasoning.

3. Approach

  • DAAAM processes RGB-D input by segmenting and tracking fragments, then reconstructing their 3D shapes and positions with Khronos at the 10Hz sensor rate.The active-window pipeline maintains temporally consistent fragments in dynamic scenes.Source: §3 · p. 3
  • The frame-selection stage accumulates observations in time windows and chooses views where each tracked fragment is visible and well positioned for semantic grounding.Its optimization balances minimizing the number of frames sent to the VLM against the quality of selected fragment-frame pairs; a binary program assigns each fragment to a visible selected frame while maximizing quality.Source: §3 · pp. 3–4
  • Selected image-fragment pairs are batch-processed by DAM to produce detailed natural-language descriptions for fragments.Batching multiple frames and masks into one tensor reduces redundant computation and improves parallelization; CLIP and sentence-embedding features are also assigned in batch.Source: §3 · p. 4
  • Place nodes capture background surfaces and topology by partitioning traversable ground into rectangles, then receive descriptions and features through projected ground-fragment annotations.The rectangles have a maximum size of 2m, and majority voting assigns descriptions and features; full-frame queries are avoided because they are often out-of-distribution for DAM.Source: §3 · pp. 4–5
  • The backend optimizes node positions and merges geometrically and semantically similar fragments and places, retaining temporal descriptions and timestamps as entity histories.It then clusters the reconciled graph into regions, assigning objects to nearby clusters and summarizing region content with a language model.Source: §3 · p. 5
  • A tool-calling agent retrieves objects by semantic search and obtains region, spatial, temporal, and agent information from the 4D scene graph to answer natural-language queries.The scene graph functions as spatio-temporal metric-semantic memory for embodied agents.Source: §3 · pp. 3, 5

4. Experiments

DAAAM is evaluated on spatio-temporal question answering, sequential task grounding, retrieval, and runtime, with tests of its benchmark performance and component choices. Results show strong long-sequence and temporal reasoning, generalization to task grounding, and real-time operation, alongside dataset caveats and a roughly 10-second worker delay.

4.1. Spatio-Temporal Question Answering

  • DAAAM performs strongly on long-sequence and temporal NaVQA questions and outperforms map- and view-based systems on OC-NaVQA, which covers sequences up to 35.8 minutes and 1.64 km.The authors also report that DAAAM struggles with some queries requiring tools beyond its available set and with small objects.Source: §4.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
  • NaVQA comparisons are complicated by test-set overlap in some prompts, noisy labels, mismatched observation windows, and spatial labels based on observation viewpoints rather than object positions.The authors re-annotate spatial queries with actual object positions and evaluate full CODa sequences in OC-NaVQA.Source: §4.1 · pp. 5–6

4.2. Sequential Task Grounding

  • On SG3D, DAAAM outperforms metric-semantic mapping systems and performs competitively with specialized task-analysis methods.The authors note a real-to-sim gap because SG3D is semi-synthetic and Describe Anything Model was trained only on real data.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3. Ablation Studies

  • DAM descriptions encoded as sentence embeddings retrieve localized subjects less accurately than CLIP alone, but combining both feature types outperforms CLIP alone.The retrieval evaluation uses five Visual Genome subsets and the refCOCOg validation set.Source: §4.3 · p. 7, Table 4 · p. 7
  • Explicit descriptions aid the retrieval-augmented agent’s compositional reasoning, especially for positional and temporal queries, while images are a better verification proxy for binary questions.The comparison uses a baseline without DAM descriptions, which instead supplies visual features and image crops.Source: §4.3 · p. 7, Table 5 · p. 7
  • Region clustering benefits all reported OC-NaVQA metrics, with the hierarchical structure particularly helpful for temporal queries involving broader context.The frame-selection quality heuristic improves spatial and question-answering accuracy, while temporal accuracy is comparable without it.Source: §4.3 · p. 7, Table 5 · p. 7

4.4. Runtime Analysis

  • DAAAM runs at CODa’s 10 Hz sensor rate, with segmentation and tracking as the main bottleneck; semantic annotation becomes a bottleneck mainly in cluttered or fast-moving scenes.Mean annotation time is 0.18 ± 0.03 seconds per fragment, and one worker is reported sufficient for a mobile ground robot workload.Source: §4.4 · p. 8, Table 6 · p. 8
  • Batching DAM inference with batch sizes of 48–128 yields an order-of-magnitude speed-up, while full-batch frame selection and semantic lifting take 1.2 ± 0.74 and 9.2 ± 1.4 seconds, respectively.The authors characterize the resulting roughly 10-second delay as a side effect of large models and argue that throughput can matter more than latency when observations are accurately summarized.Source: §4.4 · p. 8, Figure 3 · p. 8

5. Limitations

DAAAM’s limitations include imperfect localized descriptions, throughput that may not suit every platform, and a description history that may not scale indefinitely.

5. Limitations

  • DAM can miss out-of-distribution objects or uncommon visual features and hallucinate toward the mean, such as predicting handles on elevator doors.The authors relate this limitation to DAM’s relatively modest 1.5M-sample training dataset.Source: §5 · p. 8
  • 5.2 new fragments per second on a desktop GPU may be too slow for dynamic aerial robots or VR headsets.The authors note this throughput suffices for a mobile ground robot.Source: §5 · p. 8
  • Maintaining every description in dynamic nodes may let memory grow indefinitely, motivating future work on bounded-memory summarization.Source: §5 · p. 8

6. Conclusion

DAAAM combines detailed semantic descriptions with geometric grounding in a real-time spatio-temporal memory framework. It achieves state-of-the-art results in question answering and sequential task grounding while operating at 10Hz and scaling to long sequences.

6. Conclusion

  • DAAAM achieves state-of-the-art results in spatio-temporal question answering and sequential task grounding.The conclusion reports real-time performance at 10Hz and scalability to sequences exceeding 35 min and 1.5 km.Source: §6 · p. 8
  • Decoupling geometric tracking from semantic annotation through optimization-based frame selection and batch inference addresses the computational constraints of large vision annotation models.This supports real-time construction of hierarchical 4D scene graphs with detailed natural-language descriptions.Source: §6 · p. 8
  • DAAAM combines detailed semantic descriptions with geometric grounding for large-scale environments.Its memory representation is a hierarchical 4D scene graph.Source: §6 · p. 8
  • The framework is presented as a practical foundation for embodied agents to understand and interact with complex, large-scale, dynamic environments over extended time horizons.The paper releases its data and code open-source.Source: §6 · p. 8

A. Place Extraction Details

Place extraction converts the local occupancy map into a graph of traversable regions. Place nodes are then projected onto the reconstructed floor and into observing frames to obtain features and semantic descriptions.

A. Place Extraction Details

  • The occupancy map is convolved with the robot’s bounding box to form a traversability field, which is partitioned into maximal rectangles containing only traversable space.A maximum side length of 2 m is imposed for approximately uniform coverage; rectangle sides are classified by bordering traversable, unknown, or nontraversable space.Source: A. Place Extraction Details · p. 13
  • Each place is represented by a rectangle’s centroid, and two places are connected when adjacent sides are labeled as traversable boundaries.The local occupancy map is maintained for 3D reconstruction in the active window of Khronos.Source: A. Place Extraction Details · p. 13
  • For semantic lifting, place nodes are projected onto the reconstructed floor and then into frames observing each point to obtain features and semantic descriptions.Source: A. Place Extraction Details · p. 13

B. Retrieval-augmented Reasoning

A tool-calling agent retrieves information from the 4D scene graph through semantic, spatial, regional, and trajectory tools. Together, these tools return descriptions and location or time information relevant to scene queries.

B. Retrieval-augmented Reasoning

  • Semantic search returns the 10 most similar fragments to a natural-language subject using cosine similarity over CLIP and sentence-embedding features.Results include descriptions, positions, and observation timelines.Source: B. Retrieval-augmented Reasoning · p. 13
  • Spatial and regional tools retrieve nearby fragments, region summaries, or objects matching a query within a specified region.Region summaries include descriptions, entry and exit positions, and times.Source: B. Retrieval-augmented Reasoning · p. 13
  • The agent-trajectory tool returns 10 equally spaced poses, including position and heading, between specified start and end positions.Source: B. Retrieval-augmented Reasoning · p. 13

C. Example of frame selection quality-score

The example applies the frame-selection heuristic to three sequential images containing car, tree, hydrant, and person fragments. With ε = 0, the heuristic excludes a frame that omits any subject and selects the remaining frame with the highest quality score.

C. Example of frame selection quality-score

Loading 2512.00565v1…