Source-linked AI summary
Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants
Jiazhou Liang, Liam Gallagher, Kiko Chen, David Guo, Armin Toroghi, Yifan Simon Liu, Scott Sanner
TL;DR
Egocentric assistants need to connect what users say with spatial evidence about objects across long histories, while existing approaches can omit needed evidence or incur high costs. SpaC-MEM stores conversational and spatial information in persistent object records, and Ego-SpaCR evaluates this task. SpaC-MEM achieves the highest overall accuracy among evaluated methods, with fewer reference tokens than native-video baselines.
Problem
Egocentric assistants must connect user-stated facts with spatial evidence about the same objects across interaction histories, while existing approaches can omit evidence needed for complete recall.
Source: §1 · pp. 1–2Method
SpaC-MEM uses an object-centric working memory to connect conversational facts and spatial evidence through persistent physical objects.
Source: §1 · p. 2Results
SpaC-MEM achieves 52.5% overall accuracy on Ego-SpaCR versus 43.1% for the strongest baseline, using about one quarter of native-video methods’ reference input tokens.
Source: §5.2 · p. 9Takeaways & Limitations
Ablations show that retaining object geometry supports answer accuracy and object recall.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Egocentric assistants must connect what users say with what they see across long interaction histories. We formalize this challenge as Spatially grounded Conversational Reasoning (SpaCR): cross-scene, recall-oriented, and counterfactual spatial queries that combine user-stated facts with geometric evidence. Direct vision-language models incur high inference costs and context limits as histories grow, while keyframe selection and retrieval can omit objects or evidence needed for complete recall. We propose Spatially grounded Conversational Memory (SpaC-MEM), an object-centric working memory that uses 3D reconstruction and segmentation to ground conversational information in persistent physical objects. It compresses multimodal histories while preserving spatial evidence and allowing object-specific facts to be updated through dialogue. We also introduce Ego-SpaCR, a benchmark comprising 620 ScanNet video sessions augmented with 95 task-oriented conversations and 3,100 evaluation queries. SpaC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall while requiring substantially fewer reference input tokens than native-video baselines. Removing 3D spatial information substantially degrades performance, highlighting the importance of preserving spatial and conversational evidence together.
1 Introduction
SpaCR asks egocentric assistants to connect user-stated facts with spatial evidence about persistent objects across interaction histories. SpaC-MEM and Ego-SpaCR address this need with an object-centric memory and benchmark, with reported accuracy and recall gains at lower token cost.
1 Introduction
- SpaCR comprises cross-scene spatial queries that combine facts established in conversation with geometric, relational, or counterfactual evidence about objects.Such queries can require accounting for all relevant objects across the interaction history.Source: §1 · p. 1
- Existing approaches may miss objects needed for complete recall, lack evolving conversational context, or lack explicit object geometry.Source: §1 · p. 2
- SpaC-MEM connects conversational and spatial information through persistent physical objects in an object-centric working memory.It preserves spatial evidence and allows conversational information to accumulate and be updated over time.Source: §1 · p. 2
- Ego-SpaCR comprises 620 real-world video sessions, 95 task-driven conversations, and 3,100 evaluation questions spanning spatial complexity and recall breadth.Source: §1 · p. 3
- SpaC-MEM improves accuracy and object recall while substantially reducing input token cost compared with in-context methods.The introduction reports the overall comparison without specifying metric values.Source: §1 · p. 3
2 Related Work
Existing benchmarks and methods address parts of egocentric video, dialogue, or spatial reasoning, but the reviewed work does not combine evolving conversational facts with cross-scene spatial reasoning over persistent objects.
2.1 Egocentric Video and Dialogue Benchmarks
- Egocentric VideoQA and 3D spatial benchmarks evaluate observations or spatial questions, while long-term dialogue and situated-dialogue benchmarks lack spatial grounding or complex cross-scene reasoning.Source: §2.1 · p. 3
2.2 Spatial Understanding with Video Input
- Native video models keep visual and conversational history in context, while query-aware frame selection prioritizes queries and visuals rather than evolving conversation.Source: §2.2 · p. 3
- Object-memory and agentic video systems retain visual or language-guided evidence but keep conversation and visual memory parallel; retrieval can miss evidence, and summaries can omit geometry.Figure 2 groups example SpaCR queries by recall type and reasoning level; unlike VideoQA, each requires conversational facts alongside spatial evidence.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
3 SPACR: Spatially Grounded Conversational Reasoning
SpaCR formalizes queries over multimodal histories that require linking conversational facts to persistent objects and evaluating spatial conditions. It distinguishes query complexity from the breadth of the target set that must be recalled.
3 SPACR: Spatially Grounded Conversational Reasoning
- Bimodal histories comprise egocentric RGB observations and dialogue; user-stated semantic facts, such as ownership, attach properties to physical objects.Source: §3 · p. 4
- Dialogue facts can change by correction, which retroactively replaces a value, or by state change, which preserves the earlier value until the update.For a lamp changing from keep to donate, its initial status resolves to donate under correction but keep under state change.Source: §3 · p. 5
- A query uses only observations available by its turn and earlier dialogue, then matches objects against conversational and spatial conditions.The framework separates spatial complexity from the recall breadth needed to recover the matching set.Source: §3 · p. 5
- Higher spatial complexity includes recovering geometric properties such as size or distance and assessing hypothetical arrangements not directly observed.The paper notes that fit questions can require geometry beyond qualitative relations.Source: §3 · p. 5
- Object grounding links an earlier conversational condition to the same physical object; qualitative relations can evaluate spatial relations among objects.A conversationally qualified anchor can be used to identify objects sharing a relation such as being on the same table.Source: §3 · p. 5
- Recall breadth is the number of target objects supporting an answer; counting requires recovering the complete matching set, while negative answers require ruling out all candidates.Source: §3 · p. 5
4 Object-Centric Spatially Grounded Conversational Memory
SpaC-MEM compresses egocentric histories into persistent object records that connect spatial observations with dialogue, then uses those records to answer spatial queries.
4 Object-Centric Spatially Grounded Conversational Memory
- SpaC-MEM compresses frame histories into persistent object records that bind spatial and conversational information for query reasoning.The method is organized as five stages, from object reconstruction through query answering.Source: §4 · pp. 5, 7
- RGB frames are reconstructed into 3D object geometry and segmented into instances without requiring depth measurements or camera poses.Each object observation stores a class label, 3D centroid, and outlier-robust axis-aligned bounding box.Source: §4 · p. 6
- Consolidation matches same-class observations by 3D-region overlap, accepts only unambiguous matches, and updates existing objects or initializes new ones.The pool is periodically refined by merging supported fragments and removing rarely observed objects.Source: §4 · p. 6
- Dialogue association grounds turns to persistent objects, retaining original wording so later turns can revise object-specific information.Association uses current and prior objects and dialogue; reference resolution can use object classes, spatial descriptions, pronouns, and established names.Source: §4 · pp. 6–7
- Records update by object while preserving scene/session organization and dialogue order; query answering gives the LLM full memory, visible objects, and the query without retrieval.Providing complete memory lets the answerer examine all stored objects when a query requires a complete set.Source: §4 · p. 7
5 Experiments
The experiments evaluate SpaC-MEM on Ego-SpaCR against several baseline families, using answer accuracy and object recall to assess performance. SpaC-MEM leads overall, while ablations examine the roles of object-centric memory, spatial evidence, and dialogue association.
5.1 Datasets and Metrics
- Ego-SpaCR is an egocentric benchmark built from ScanNet, covering four defined SpaCR properties; real observations, annotated object identities, and geometry enable verifiable spatial evidence and reconstruction-error analysis.Queries combine conversational conditions with spatial operations, represented by deterministic programs that produce ground-truth answers from facts and ScanNet geometry. For count queries near geometric boundaries, a bounded answer interval accounts for uncertainty in the annotated mesh.Source: §5.1 · pp. 7–8
- Answer accuracy is the primary metric, while object recall measures the fraction of required objects recovered for nonempty count queries.Object descriptions are mapped to physical objects to assess both selected-object correctness and support for counts.Source: §5.1 · p. 8
5.2 Results
- 52.5% overall accuracy for SpaC-MEM exceeds the strongest baseline’s 43.1%; on C4, it scores 61.4% versus 37.3% for the strongest baselines.SpaC-MEM uses about one quarter of the reference input tokens required by native-video methods.Source: Table 2 · p. 9, §5.2 · p. 9
- Oracle boxes yield only marginal gains, while oracle geometric measurements improve C3 but provide little improvement on C4 counterfactual queries.The results suggest that consistent spatial representation across observations, rather than object detection or geometric measurement alone, is the main limitation of existing baselines relative to SpaC-MEM.Source: §5.2 · pp. 9–10, Figure 6 · p. 10
- SpaC-MEM uses 4.7× fewer input tokens than per-frame memory, and removing spatial detail reduces accuracy from 51.5% to 34.9%.Adding object RGB views recovers little of the loss; LLM dialogue association retains 89% of the accuracy obtained with human-annotated associations.Source: §5.2 · p. 10, Figure 7 · p. 10
6 Conclusion
The conclusion presents Ego-SpaCR and SpaC-MEM as connecting egocentric observations with evolving dialogue through persistent object records. SpaC-MEM achieves the highest overall answer accuracy among evaluated methods while using fewer reference input tokens than native-video baselines.
6 Conclusion
- Ego-SpaCR evaluates spatially grounded conversational reasoning across egocentric observations and evolving dialogue, while SpaC-MEM connects those histories through persistent object records..Source: §6 · p. 10
- SpaC-MEM achieves the highest overall answer accuracy among evaluated methods while requiring fewer reference input tokens than native-video baselines.The conclusion attributes support for answer accuracy and object recall to retaining object geometry.Source: §6 · pp. 10–11
A Related Work Comparison
- The related-work comparison organizes benchmarks by visual input, conversation, object identity, and evaluation, distinguishing explicit support from related or limited support.Its note says static scene annotations alone do not count as explicit cross-view object continuity.Source: Table 3 · p. 17, A Related Work Comparison · p. 17
B Additional Benchmark Details
The appendix details how ScanNet observations and dialogue are composed into benchmark episodes, how object-specific facts and references are tracked, and how queries and conversations are generated and checked.
B.1 ScanNet observations and episode composition
- ScanNet RGB-D frames, camera poses, instance annotations, and 3D meshes provide observations, persistent object identities, visibility, and spatial measurements for the benchmark.The benchmark composes visits across selected scenes, keeps revisited objects identifiable, and assigns all clips from a physical scene to one split.Source: §B.1 · p. 17
B.2 Tasks, personas, and conversational state
- Five everyday tasks and compatible personas guide dialogue, while object facts are recorded as object-property-value triples with temporal scope and later corrections or state changes.References resolve through context, appearance, spatial relations, established names, or clarification; names can carry across later visits.Source: §B.2 · p. 18, Table 4 · p. 18
B.3 Conversation generation
- Queries are selected only when their semantic prerequisites and spatial evidence can be available before the question, with required facts scheduled in advance.Each query has a deterministic program, temporal scope, and placement window where its answer is stable.Source: §B.3 · p. 18
- An LLM Writer generates scene-level dialogue batches from task, persona, prior conversation, observations, references, and facts, while query answers are withheld.An independent Reader reconstructs stated facts and updates turn by turn from the realized visual and dialogue history.Source: §B.3 · p. 19, Table 5 · p. 19
B.4 Query construction and recall
- Queries include cross-scene aggregates and counterfactual viewpoints or placements; recall-oriented questions require the complete qualifying object set, retained alongside the answer for checking.Qualitative and geometric predicates use objects and measurements from the same source scene.Source: §B.4 · p. 20
B.5 Ground truth and answer evaluation
The evaluation defines canonical answers from the dialogue and spatial evidence available before each query, then checks question validity through automated execution and human review. It also specifies control baselines and a model-independent measure of reference input context.
B.5 Ground truth and answer evaluation
- Canonical answers execute each question program over the pre-query dialogue facts and ScanNet spatial evidence, recording qualifying objects and supporting spatial measurements.For boundary-sensitive counts, candidates within 0.04 m of a measured boundary contribute to an accepted count interval whose width is bounded to two.Source: §B.5 · p. 20
B.6 Verification and quality review
- Questions are retained only when required facts, references, objects, and spatial evidence are available at the realized query time and execution reproduces the planned answer.Human review separately checks semantic grounding, reference resolution, temporal scope, conversational plausibility, and answer correctness.Source: §B.6 · p. 20
C Control Baselines
- Random predicts mean canonical count or Boolean answers and samples available objects, while Conversation Only answers from timestamped pre-query dialogue without images or frame indices.The Random control uses seed 20260925 and a separate deterministic draw per query.Source: C Control Baselines · pp. 20–21
D Reference Input Cost
- Reference input cost counts the answer model’s prompt text and supplied RGB frames using a shared tokenizer and 300 reference tokens per frame.The offline mean is measured over scored requests and excludes pipeline compute such as retrieval embeddings, memory updates, tool calls, and output tokens.Source: D Reference Input Cost · p. 21
E Additional Method Details
SpaC-MEM reconstructs and consolidates persistent 3D objects, then associates dialogue with those objects to form an object-centered working memory. The memory retains spatial records and verbatim object-linked conversational turns.
E.1 Object reconstruction and consolidation
- RGB frames are placed in a shared, approximately metric scene coordinate system, with axes aligned to dominant planes and only high-confidence depth pixels lifted into 3D.Wall and floor are excluded as structural surfaces rather than objects.Source: §E.1 · p. 21
- Mask pixels are retained when their depth falls within a widened central interior range, preserving matching edge pixels such as chair legs; centroids and robust axis-aligned boxes summarize the lifted points.The box computation excludes the most extreme points to reduce the effect of stray points.Source: §E.1 · p. 22
- Observations are matched to same-class scene objects by 3D point overlap within 10 cm, with joint frame matching preventing one object from taking multiple observations.Matched observations update the accumulated points and class evidence; runner-up classes can be retained.Source: §E.1 · p. 22
- Pool refinement merges substantially overlapping objects only when they never co-occur and share a segmentation track or similar appearance, then removes isolated points from confirmed objects.Each scene keeps its own levelled coordinate frame; spatial predicates are evaluated within scenes and cross-scene queries aggregate results.Source: §E.1 · p. 22
E.2 Dialogue association and working memory
- The association model links preceding dialogue turns to observed objects using class labels, geometry, and nearby conversational context, assigning only identifiable subject-role references.Working memory stores one record per persistent object with spatial details and verbatim associated turns, marking the phrase that refers to that object.Source: §E.2 · p. 22
F VLM Judge for Object Identification
The object-identification judge evaluates whether an answer uniquely resolves to the target among objects satisfying the conversation-derived condition. It uses candidate evidence and does not apply the query’s selection rule on the model’s behalf.
F VLM Judge for Object Identification
- The judge receives a fixed candidate set satisfying the conversation-derived condition, with candidates uniformly presented using a selected frame, boxed object, shuffled letter, and no indication of the target.The candidate set is constructed before method evaluation and always includes the ground-truth target.Source: F VLM Judge for Object Identification · p. 23
- Each candidate includes conversation facts and a compact ground-truth scene graph, allowing location descriptions to be checked even when a referenced neighbor is outside the selected frame.The graph includes nearby objects within 3 m and on/supports relations; the benchmark defines near as ≤1.5 m and far as ≥3 m.Source: F VLM Judge for Object Identification · p. 23
- The judge resolves an answer against every candidate using its stated class, appearance, names, facts, and additional location claims, without applying the query’s selection rule.Correctness requires the resolved candidate set to contain exactly the target; three judge calls at temperature 0 determine the majority verdict.Source: F VLM Judge for Object Identification · p. 23
- A human annotator independently verified 500 sampled identification answers drawn across 12 baselines using the same candidate presentation.Source: F VLM Judge for Object Identification · p. 23
- Table 6 reports answer accuracy for selected baselines and SpaC-MEM with Wilson 95% confidence-interval half-widths, marking the best and second-best values per column.The supplied table caption does not provide cell values.Source: Table 6 · p. 24
G Additional Experimental Results
Figure 8 reports object-recall ablations: SpaC-MEM retains about 70% recall as history grows, while per-frame memory degrades at 4.7× the token cost; spatial evidence and dialogue association also affect recall.
G Additional Experimental Results
- About 70% recall is retained by SpaC-MEM as history grows, while per-frame memory degrades at 4.7× the token cost; spatial evidence drives recall on spatial queries, and LLM-predicted association retains most human-annotated GT recall.Source: Figure 8 · p. 25
H.1 Object recall in the ablations
The ablations show how object-centric memory and preserved spatial evidence affect object recall, while scaling and reconstruction analyses examine recall across memory sizes and reconstructed-object fidelity.
H.1 Object recall in the ablations
- SpaC-MEM maintains roughly 70% object recall as history grows with 4.7× fewer input tokens, while per-frame recall with Luna falls from 73% to 35% after 11 sessions.Source: §H.1 · p. 25
- Preserving 3D records raises object recall from 46.2% to 69.7% overall and from 33.6% to 76.5% on C4.Source: §H.1 · p. 25
- Automatic dialogue associations retain 87% and 85% of the recall achieved with human annotations for Luna and GLM-5.3 Flash, respectively.Source: §H.1 · p. 25
H.2 Scaling with memory size
- Across the range covering 95% of queries, object recall remains near 70% for both answering models as accuracy changes gradually with working-memory size.Source: §H.2 · p. 25, Figure 9 · p. 26
H.3 Fidelity of reconstructed objects
- The reconstruction evaluation covers 39 scenes and 864 annotated objects, matching records one-to-one by centroid distance and requiring class agreement for semantic recall.Source: §H.3 · p. 25, Table 7 · p. 26
- At 0.5 m, the persistent pool recovers 91.1% of annotated objects, including 82.2% with the correct class, with a 16.2 cm median centroid error among matches.Source: §H.3 · p. 25, Table 7 · p. 26