PaperlayerView exact revision on arXiv

Source-linked AI summary

Episodic Memory Question Answering

Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, Devi Parikh

arXiv:2205.01652v1cs.CVcs.AI

TL;DR

Egocentric assistants need to answer questions about what they observed and where, but existing embodied question-answering approaches do not retain scene information across questions. This paper introduces EMQA and a top-down episodic-memory model, which outperforms baselines and shows resilience to noisy inputs.

  • Problem

    Egocentric assistants need to answer post-tour questions with localized responses, whereas embodied question-answering approaches described here use clean-slate episodes and language-token answers.

    Source: §1 · pp. 1–2

  • Method

    EMQA pairs grounded questions about guided tours with an allocentric top-down semantic scene memory augmented by observation-time information and a LingUNet question-answering model.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    The episodic scene-memory model outperforms strong baselines and is more resilient to noisy pose perturbations, with tests also covering imperfect depth and camera jitter.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors present the results as promising evidence for future research toward deploying egocentric assistants in real-world AR devices.

    Source: §7 · p. 8

    Static scene maps constrain the approach to objects whose positions remain largely fixed; more frequent tour resampling could better approximate the current environment state.

    Source: §6 · p. 8

Abstract

from arXiv · show

Egocentric augmented reality devices such as wearable glasses passively capture visual data as a human wearer tours a home environment. We envision a scenario wherein the human communicates with an AI agent powering such a device by asking questions (e.g., where did you last see my keys?). In order to succeed at this task, the egocentric AI assistant must (1) construct semantically rich and efficient scene memories that encode spatio-temporal information about objects seen during the tour and (2) possess the ability to understand the question and ground its answer into the semantic memory representation. Towards that end, we introduce (1) a new task - Episodic Memory Question Answering (EMQA) wherein an egocentric AI assistant is provided with a video sequence (the tour) and a question as an input and is asked to localize its answer to the question within the tour, (2) a dataset of grounded questions designed to probe the agent's spatio-temporal understanding of the tour, and (3) a model for the task that encodes the scene as an allocentric, top-down semantic feature map and grounds the question into the map to localize the answer. We show that our choice of episodic scene memory outperforms naive, off-the-shelf solutions for the task as well as a host of very competitive baselines and is robust to noise in depth, pose as well as camera jitter. The project page can be found at: https://samyak-268.github.io/emqa .

1. Introduction

Egocentric assistants need to remember what they saw and where, then ground follow-up answers in that memory; EMQA frames this as localizing answers to questions about a guided indoor tour. The paper introduces a dataset and a top-down episodic-memory model, reporting advantages over several baselines and resilience to noisy inputs.

1. Introduction

  • EMQA asks an assistant to localize answers to post-hoc questions about a guided indoor tour, using a scene memory that can support multiple follow-up questions.The input may be a sequence of RGB-D images with oracle poses; answers can be localized in the tour frames or on a top-down map.Source: §1 · p. 2
  • The dataset probes spatial understanding and temporal reasoning with questions such as where an object was first or last seen.The model builds allocentric top-down semantic scene representations and augments them with information about when locations were observed.Source: §1 · p. 2
  • The proposed model outperforms language-only baselines by ∼150% and frame-by-frame localization solutions by 37%, as well as memory representations that aggregate observation features.The compared memory baselines use averaging, GRU, or context-conditioned attention.Source: §1 · p. 2
  • The approach is more resilient than baselines to noisy pose perturbations and is tested under imperfect depth, pose, and camera jitter in a real-world RGB-D setting.The paper also reports qualitative zero-shot transfer to that setting.Source: §1 · p. 2

2. Related Work

EMQA differs from prior embodied and video question-answering settings by using a persistent scene memory and grounding answers in the tour or floorplan. Its dataset derives grounded spatial and temporal questions from observed objects in indoor exploration tours.

2. Related Work

  • Unlike embodied question-answering episodes that start clean-slate and predict language tokens, EMQA shares persistent scene memory across questions and localizes answers in the environment.Compared with VideoQA, it uses indoor navigation videos without per-frame subtitles or similar annotations and supports floorplan grounding.Source: §2 · pp. 2–3

3. EMQA Dataset: Questions Grounded in Scene Tours

  • The dataset uses manually recorded, coverage-optimized Matterport3D exploration paths with egocentric RGB-D observations and ground-truth pose.The scans cover 90 indoor environments, and paths average 2,500 steps.Source: §3 · p. 3
  • Ground-truth top-down semantic maps are filtered to locations observed during the tour, so questions are generated from witnessed objects rather than every object in the scene.Depth maps are projected onto the top-down map at each tour step to create the observed-location masks.Source: §3 · p. 3, Figure 2 · p. 4
  • Spatial questions ask where an object category was seen, while temporal questions select the instance first or last observed during the tour.An instance counts as seen when the observed mask covers more than 10% of it at a time step; first- and last-seen instances can coincide on looping tours.Source: §3 · pp. 3–4
  • Train, validation, and test splits use mutually exclusive scenes, and randomly sampled 20-step short-tour segments supplement training while the task remains defined on full tours.The dataset statistics cover both short and full tours.Source: §3 · p. 4

4. Models

The model builds a top-down semantic memory by projecting egocentric observations into a floorplan and accumulating them across the tour. It adds temporal observations and uses a language-conditioned model to localize answers on the map.

4. Models

  • Egocentric semantic features are projected onto a top-down floorplan and accumulated with a GRU into a spatial episodic-memory tensor.The projected cells correspond to 2 cm × 2 cm metric regions and encode object semantics.Source: §4 · pp. 4–5
  • The 2D scene features model inter-object spatial relationships through geometrically consistent projections and avoid the memory constraints of voxel-based representations.The representation is allocentric and top-down.Source: §4 · pp. 4–5
  • The spatiotemporal memory adds channel-stacked observations of when each metric cell was seen during the tour.This extends the scene representation with temporal information for tour reasoning.Source: §4 · p. 5
  • A LingUNet-based question-answering model combines the scene memory and question representation to produce a spatial answer map over the floorplan.The model uses an LSTM question encoder and processes the resulting spatial feature map into answer scores.Source: §4 · p. 5, Figure 3 · p. 5
  • Training proceeds in stages: semantic segmentation trains the visual encoder, top-down semantic maps train the scene encoder, and answer maps train the question-answering model.The answer-map objective uses per-pixel binary cross entropy with focal-loss weighting for class imbalance.Source: §4 · p. 6

5. Baselines

The baselines test whether questions can be answered from language alone, egocentric segmentation, top-down semantic decoding, or buffers of tour features. Buffer-based approaches decode a scene embedding into a top-down answer heatmap, whereas the proposed model explicitly projects egocentric features geometrically.

5. Baselines

  • Language-only, egocentric-segmentation, and top-down-decoding baselines test alternatives that omit episodic scene memory or temporal features.LangOnly keeps temporal features but drops scene memory; EgoSemSeg predicts the queried object from tour frames, while SMNetDecoder predicts top-down semantic labels before extracting that object.Source: §5 · p. 6
  • Three buffer baselines condense RedNet tour features by averaging, GRU, or question-conditioned scaled dot-product attention.The resulting variants are EgoBuffer-Avg, EgoBuffer-GRU, and EgoBuffer-Attn.Source: §5 · p. 6
  • Buffer-based baselines decode a 1-D scene embedding into a top-down 2-D answer heatmap, implicitly converting egocentric observations into an allocentric map.The proposed model instead builds this transformation in through geometrically consistent projections of egocentric features.Source: §5 · p. 6

6. Experimental Results

The proposed model outperforms SMNetDecoder and feature-buffer memory baselines, with temporal features improving spatio-temporal localization. Under noisy pose and real-world inputs, performance degrades but shows greater resilience than the best baseline, and retraining recovers some lost performance.

Quantitative Results

  • The model gains 8.2% in IoU and 42% in recall over SMNetDecoder, while EgoSemSeg performs worse than SMNetDecoder.The authors attribute the gains over SMNetDecoder to temporal information and LingUNet-based question grounding.Source: §6 · p. 7
  • Feature-buffer scene-memory baselines perform poorly, while the proposed approach builds semantic maps that preserve spatial and temporal information for answer localization.The paper argues that compressed one-dimensional scene representations are inadequate for this task.Source: §6 · p. 7

Temporal features help

  • Adding temporal features yields a 24% relative IoU improvement on spatio-temporal questions, with no significant impact on spatial-question metrics.The model also qualitatively distinguishes all-, first-, and last-seen table instances.Source: §6 · p. 7

Sim2Real Robustness

  • In zero-shot tests on real-world RGB-D video, the model grounds answers and reasonably distinguishes first- from last-seen instances despite noisy depth and pose and camera jitter.The evaluation uses RGB-D SLAM benchmark data without fine-tuning.Source: §6 · pp. 7–8

Noisy pose

  • The proposed model's IoU drops 29% under noisy pose, versus a 36% drop for the best-performing baseline, and retraining increases IoU and precision across all three noise models.With oracle-localization training, scene-representation quality and task metrics decline as noise severity and type vary.Source: §6 · p. 8, Figure 5 · p. 8

Limitations

  • Static scene maps constrain the setup to questions about objects whose positions remain largely fixed.The authors suggest periodically resampling tours to better approximate the current environment state.Source: §6 · p. 8

7. Conclusion

The paper studies question answering in 3D environments for egocentric personal AI assistants and proposes semantic scene features as episodic memory. The approach answers scene questions and outperforms strong baselines, with robustness investigations providing promising evidence toward real-world AR deployment.

Conclusion

  • The paper studies question answering in 3D environments with the goal of supporting egocentric personal AI assistants.It proposes semantic feature representations of scenes as episodic memory.Source: §7 · p. 8
  • The authors report that episodic semantic scene representations enable effective scene-question answering and outperform strong baselines.Source: §7 · p. 8
  • Robustness investigations across input noise provide promising evidence for future research toward deploying such agents in egocentric AR devices.Source: §7 · p. 8

8. Additional Dataset Details

The supplementary dataset details describe the object categories and spatial distribution represented in the questions. The examples span object locations and sizes, with locations distributed across the map rather than concentrated in particular areas.

Additional Dataset Details

  • The dataset uses 12 of Matterport3D's 40 object categories, selecting the most common types and excluding thin-array categories such as walls and curtains.The selected categories include shelving, fireplace, bed, table, plant, drawers, counter, cabinet, cushion, sink, sofa, and chair.Source: §8 · p. 10
  • Additional qualitative examples show questions alongside their ground-truth top-down answer maps.Source: §8 · p. 10
  • Across short-tour splits, the six illustrated object types are distributed across 250 × 250 maps, with circle centers marking instance locations and radii marking object size.Maps use 2 cm × 2 cm cells, so pixel coverage correlates with real-world object size.Source: §8 · p. 10
  • Object locations are broadly distributed across the maps, and questions cover a range of object sizes, from cushions to beds.Mean locations for nearly all 12 categories are close to the map center.Source: §8 · p. 10

9. Additional Model+Training Details

The model stores tour timing in top-down map cells and uses question-conditioned LingUNet decoding to produce answer maps. Additional baselines and training details specify how tour features are aggregated and optimized.

9. Additional Model+Training Details

  • Each top-down cell receives a 20-bit multi-hot vector marking which of 20 tour segments observed that location.The tour is divided into 20 equal-length segments, and each segment produces a binary map of observed cells.Source: §9 · pp. 11–12
  • LingUNet combines a 256×250×250 spatio-temporal memory tensor with a 64-dimensional question embedding to predict a 250×250 answer map.Question-conditioned filters create language-conditioned skip connections in a three-layer encoder-decoder.Source: §9 · p. 12
  • Training uses per-map-pixel focal loss with gamma 2.0, Adam at 2e−4 learning rate, and weight decay 4e−4.Singleton question batches accumulate gradients across four iterations before each weight update.Source: §9 · p. 12
  • EgoBuffer-GRU uses the final hidden state over flattened per-step features, while EgoBuffer-Attn uses question-conditioned scaled dot-product attention.EgoBuffer-Avg instead averages per-step feature volumes before a convolutional network produces a tour representation.Source: §9 · pp. 12–13
  • The EgoBuffer baselines and proposed model train on fixed-size 20-step tours, then combine segment outputs with pose information for full-tour evaluation.For each 20-step segment, EgoBuffer baselines generate a 250×250 output map.Source: §9 · p. 13

10. Additional Quantitative EMQA Results

The appendix reports additional results on short tours while emphasizing that the paper’s evaluation focus is full-scale exploration tours and maps. The short-tour results follow the trends discussed for the main-paper evaluations.

10. Additional Quantitative EMQA Results

  • Short-tour quantitative and qualitative results are provided for completeness alongside the main paper’s full-scale evaluation.The full-scale exploration tours and maps remain the subject of the paper’s evaluations and analysis.Source: §10 · p. 13
  • The paper’s evaluation focus is full-scale exploration tours and maps, although models are trained using sampled 20-step short tours.Full-scale evaluation tests generalization from those short-tour training examples to entire exploration trajectories.Source: §10 · p. 13
  • Table 2 reports EMQA results for the proposed model and baselines on short tours in top-down-map and egocentric-pixel output spaces.The passage states that the trends discussed for the main paper also hold for these short-tour results, but supplies no cell values.Source: §10 · p. 13, Table 2 · p. 14

11. Additional Qualitative EMQA Results

The appendix adds qualitative examples of the proposed EMQA model’s predictions on both short and full tours.

11. Additional Qualitative EMQA Results

  • Additional qualitative prediction examples for the EMQA model cover both short and full tours.The examples are shown in Figures 10 and 11, respectively.Source: §11 · p. 14

12. Sim2Real: Real-world RGBD Results

The paper presents zero-shot real-world RGB-D evaluation using a pretrained scene-memory model and question-answering module, without fine-tuning on that dataset. Qualitative chair examples show first-, last-, and general-location predictions, alongside a missed instance.

12. Sim2Real: Real-world RGBD Results

  • Zero-shot evaluation uses pretrained SMNet and LingUNet modules, with no component fine-tuned on the real-world dataset.The video is subsampled by selecting every tenth frame, and blurry images are discarded using a Laplacian-variance threshold of 100.Source: §12 · p. 14
  • For “where did you see the chair?”, the model detects three chairs but misses the fourth chair bearing a stuffed animal.The real-world RGB-D examples include imperfect depth and pose and camera jitter.Source: §12 · p. 14
  • For first- and last-seen chair questions, the bottom two chairs are detected in both predictions, while the third chair appears only in the first-seen prediction.The third chair is not strictly the first chair instance in the video, although it appears near the beginning.Source: §12 · p. 14

13. Sim2Real: Noisy Pose Experiments

The experiments examine how independent and cumulative pose noise affect scene-memory construction and question answering, and whether retraining on noisy poses helps. Retraining the visual-odometry model for the guided-tour setup also improves its trajectory estimates.

13.1. Noise Models

  • Independent noise perturbs each ground-truth tour pose, whereas cumulative noise integrates per-step visual-odometry estimates, allowing errors to cascade along the trajectory.Independent noise uses linear and rotational actuation models at 0.5× and 1× intensity.Source: §13.1 · pp. 14–15

13.2. Visual Odometry Models

  • Retraining the VO model on Matterport3D under EMQA’s actuation specifications reduced trajectory deviation by approximately 6× versus the pretrained model.The pretrained model used different scenes and actuation specifications; the table compares average absolute and relative pose differences.Source: §13.2 · pp. 15–16, Figure 12 · p. 16, Table 3 · p. 17
  • Trajectory deviation increased approximately 1.3× with stronger independent noise and 15× when moving from independent to cumulative noise.The table reports average pose differences from ground truth at each step.Source: §13.2 · pp. 16–17, Table 3 · p. 17

13.3. Evaluating EMQA Models on Noisy Pose Data

  • Across the three evaluations, noisy-pose inputs reduced performance in models trained with oracle poses, while retraining recovered some of the drop and sharpened map boundaries with less label splatter.Figure 14 compares ground-truth training and evaluation, evaluation under noise, and retraining with noisy poses.Source: §13.3 · p. 17, Figure 14 · p. 18
  • Our method’s IoU dropped 39% under noisy-pose evaluation and increased 10% after retraining; SMNet map IoU dropped 43% and gained 6%, while SMNetDecoder QA IoU dropped 41% and gained 19%.The comparison evaluates ground-truth-trained models on noisy poses, then evaluates models retrained on noisy poses; Figure 13 presents these three experiments.Source: §13.3 · p. 17, Figure 13 · p. 18

Related papers

Loading 2205.01652v1…