PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, James Fort, Richard Newcombe, Hyo Jin Kim, Mi Zhang

arXiv:2606.00825v1cs.CVcs.ETcs.HCcs.MA

TL;DR

Existing egocentric benchmarks capture short-term perception rather than the long-horizon, multimodal memory needs of AI-glasses users. SuperMemory-VQA evaluates these needs with grounded questions, and finds current systems remain unreliable at answerability, long temporal gaps, and integrating evidence across moments.

  • Problem

    Existing egocentric benchmarks emphasize short clips and generic perception, leaving practical, long-horizon memory questions over continuous multimodal streams insufficiently evaluated.

  • Method

    SuperMemory-VQA combines multimodal AI-glasses recordings with temporally grounded questions, evidence annotations, ordered answer choices, and an explicit unanswerable option.

  • Results

    Current video-understanding and retrieval-augmented systems remain unreliable, struggling with answerability, long temporal gaps, and multi-moment evidence integration.

  • Takeaways & Limitations

    The benchmark advances evaluation toward grounded, situated memory systems that answer only when available evidence is sufficient.

  • Takeaways & Limitations

    Long-horizon summarization can lose exact conversational attributes, causing plausible substitutions from adjacent context that are difficult to detect.

Abstract

from arXiv · show

AI glasses present a compelling platform for AI agents to serve as personalized memory assistants. To be genuinely useful, such systems must move beyond short-term video comprehension and address memory gaps that humans experience for practical, personal, or social purposes over longitudinal egocentric video streams. However, existing egocentric datasets predominantly focus on action recognition or generic QAs from short clips, measuring perceptual capabilities rather than realistic human memory needs. We introduce SuperMemory-VQA, an egocentric visual question answering (VQA) dataset for evaluating AI assistants on practical, long-horizon memory tasks. It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories. Through a human-verified annotation pipeline, we construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval. Each question is posed as multiple-choice with an explicit "unanswerable" option to test hallucination robustness. Benchmarking leading agentic frameworks and LLM backbones reveals that existing systems remain far from reliable on real-world memory tasks, highlighting the need for new architectures for grounded AI memory that can answer only when evidence is sufficient. A participant survey further supports that our questions are realistic, useful, and aligned with everyday memory needs.

1 Introduction

SuperMemory-VQA introduces a multimodal, long-horizon egocentric VQA benchmark designed around realistic memory questions for AI-glasses assistants. Its human-in-the-loop construction, unanswerable options, benchmarking, and survey expose reliability gaps while supporting the dataset’s practical relevance.

  • Motivation: AI-glasses agents could serve as personalized memory systems, but useful assistance requires processing continuous multimodal streams over extended periods.They are intended to help users locate misplaced items, revisit conversations, and reconstruct past events.
  • Benchmark design: The benchmark emphasizes comprehensive memory tasks, long-horizon context, multi-evidence reasoning, and realistic, context-grounded questions.These properties address limitations of prior benchmarks centered on short-term visual perception or lacking systematic user-centered memory evaluation.
  • Annotation: A scalable human-in-the-loop pipeline generates grounded question-answer pairs, applies automated checks, and adds human verification and refinement.Each multiple-choice question includes accurate, vague, incorrect, and unanswerable options to evaluate answer quality and hallucination robustness.
  • Findings: Current agentic frameworks and vision-language models remain far from reliable, struggling with answerability detection, long temporal gaps, and evidence integration across moments.A participant survey found the questions practical, relevant, and aligned with everyday AI memory needs.
  • Dataset: SuperMemory-VQA contains 52.9 hours of multimodal egocentric activity and 4,853 question-answer pairs for practical, long-horizon memory tasks.Recordings include synchronized RGB video, spatial audio, eye gaze, IMU, and SLAM trajectories.

2 Related Work

Prior egocentric datasets evolved from small, gaze-focused collections to large multimodal recordings, while long-video understanding increasingly uses retrieval over captions, frames, and multimodal evidence. These approaches expand available context but face persistent challenges in scalable frame-level indexing and retrieval.

  • Egocentric Multimodal Datasets: Egocentric datasets progressed from small eye-tracking collections to hundreds or thousands of hours, with Project Aria adding synchronized RGB, gaze, spatial audio, IMU, and 3D scene context.Early datasets were narrow, whereas later efforts primarily remained centered on RGB video before broader sensor suites emerged.
  • Long Video Understanding: Long-video systems use retrieval-augmented generation over captions, frames, ASR, OCR, and detections to extend context and preserve visual details.Video-RAG jointly retrieves these modalities, but frame-level retrieval remains difficult to index and scale.

3 SuperMemory-VQA Dataset

SuperMemory-VQA is a multimodal, long-horizon egocentric memory benchmark built from 52.9 hours of recordings and 4,853 human-verified QA pairs. Its six memory tasks, natural queries, multi-evidence grounding, and calibrated unanswerable option target practical memory assistance and hallucination robustness.

  • Dataset and Collection: 52.9 hours of everyday egocentric recordings from ten participants provide synchronized RGB video, SLAM, eye-tracking, and seven-channel audio across indoor and outdoor activities.Participants wore Gen 1 Meta Aria Glasses; audio is released as privacy-filtered WhisperX transcripts, with faces and license plates obscured.
  • Memory Tasks: The six tasks cover object locations, spoken facts and commitments, visual details, relational retrieval, chronological event sequencing, and stated or implied future intentions.These dimensions correspond to episodic, conversational, semantic, relational, temporal, procedural, and prospective memory aspects.
  • Benchmark Design: Questions use conversational phrasing and hours- or days-long recordings, often requiring retrieval of sparse evidence across disjoint moments.The benchmark emphasizes intent inference, temporal references, retrieval-augmented reasoning, and temporal abstraction.
  • Benchmark Design: Multimodal grounding aligns video, audio, gaze, motion, and spatial context to track actions, object states, and user intent over time.Auditory cues can localize events before visual confirmation, such as steam venting from an Instant Pot.
  • Benchmark Design: The benchmark uses ordered answer choices distinguishing correct, vague, wrong, and unanswerable responses to evaluate epistemic calibration and hallucination avoidance.The ranking Correct > Vague > Wrong also supports alignment feedback such as DPO.
  • Dataset Composition: 4,853 QA pairs are balanced across six memory tasks while preserving natural variation in question complexity.Object & Location Memory and Conversational Memory comprise approximately 19% and 18%, respectively; the remaining tasks are Visual Scene Recall, In-Context Retrieval, Timeline Reconstruction, and Intent Recall.

4 Experimental Setup

The experiments compare Video-RAG and EgoButler across diverse open- and closed-source VLMs under causally constrained, reproducible settings, using answerability, accuracy, and ranking metrics.

  • Frameworks: Video-RAG retrieves ASR, OCR, and object-detection evidence through parallel FAISS databases before VLM answer generation.It is training-free and single-turn, augmenting the VLM with auxiliary text extracted from the source video.
  • Frameworks: EgoButler builds a hierarchical memory bank from dense visual–audio captions and recursively summarized hour- and day-level digests.EgoRAG uses coarse-to-fine temporal localization, retrieving summaries before narrowing to clips.
  • Models and implementation: The benchmark evaluates six open-source and four closed-source VLMs, including Qwen-3-VL, InternVL-3.5, Gemma, Gemini, and GPT-5.4 variants.Open-source models run on 4×A100 GPUs, while closed-source models use official APIs.
  • Models and implementation: Video-RAG partitions histories into 30-minute shards, fans retrieval across preceding shards, and supplies 32 frames with merged auxiliary texts.EgoButler generates clip captions over 30-second windows at 1 fps and summarizes them at hour and day levels.
  • Evaluation: Both frameworks receive the question and four choices with video truncated at the question end time, and performance is measured by Ans-F1, QA-Acc, and QA-MRR.Ans-F1 evaluates answerability decisions, QA-Acc measures four-way choice accuracy, and QA-MRR rewards ranking the correct answer higher.

5 Benchmarking Results

Video-RAG improves answerability detection and retrieval-based memory coverage over EgoButler, but benchmark performance remains limited by accuracy gaps, excessive abstention, and failures in precise long-horizon evidence use. Gemini-3-Flash is the strongest configuration, while model size alone does not reliably predict performance.

  • Overall benchmark performance: 61.0% QA-Acc is achieved by Gemini-3-Flash with Video-RAG, despite 83.9% Ans-F1 and 76.0% QA-MRR, exposing a substantial answerability-to-accuracy gap.Reliable answers still require precise multimodal retrieval, interpretation, distractor discrimination, and appropriate abstention.
  • Framework comparison: Video-RAG raises average Ans-F1 from 51.5% to 70.5%, QA-Acc from 41.4% to 46.6%, and QA-MRR from 62.8% to 66.4% over EgoButler.It achieves higher Ans-F1 for every model and higher QA-MRR for nine of ten models, with one tie.
  • Model comparisons: Closed-source models average 76.8% Ans-F1, 53.6% QA-Acc, and 70.9% QA-MRR under Video-RAG, versus 66.4%, 41.9%, and 63.4% for open-source models.However, performance is not monotonic with model size: Qwen-3-VL 30B has 56.6% Video-RAG Ans-F1 versus 75.0% for Qwen-3-VL 8B, while Gemma 31B improves over E4B IT on every metric.
  • Task-level and system comparisons: Video-RAG provides more balanced coverage of memory tasks than EgoButler, especially when evidence must be linked across time, while VideoAgent underperforms both across most dimensions.VideoAgent’s added computational cost does not translate into improved SuperMemory-VQA performance, with its largest gaps in Conversational Memory, Intent Recall, and Timeline Reconstruction.
  • Reliability analysis: 39.9% of answerable cases are wrongly abstained by Gemini-3-Flash, while several open-source models wrongly abstain on more than 70% of answerable questions.Gemini-3-Flash answers correctly on only 42.9% of answerable cases, making excessive abstention a dominant reliability failure.
  • Qualitative error analysis: Qualitative failures involve retrieving the wrong moment, losing detail during summarization, conflating nearby conversational facts, and missing small visual or OCR-style evidence.Sparse, localized evidence can make answers nearly direct once retrieved, but harder cases require disambiguation, counting, temporal ordering, premise checking, and preserving exact conversational attributes.

6 Conclusion

SuperMemory-VQA is an egocentric benchmark for long-horizon memory that combines multimodal AR-glass recordings with temporally grounded questions and evidence annotations. Its preliminary evaluation finds current systems unreliable on answerability, long temporal gaps, and multi-moment evidence integration, while participants confirm practical daily relevance.

  • SuperMemory-VQA introduces an egocentric VQA benchmark for long-horizon memory in AI assistant settings.
  • The benchmark combines multimodal AR-glass recordings with temporally grounded questions, evidence annotations, and ordered choices distinguishing accurate, vague, incorrect, and unanswerable responses.
  • Current video-understanding and retrieval-augmented systems remain unreliable on answerability, long temporal gaps, and multi-moment evidence integration.
  • Participant feedback confirms the benchmark’s practical relevance for daily use.

A Task Description Supplementary Information · A.1 Task Taxonomy

SuperMemory-VQA organizes its question-answer pairs into six functional tasks designed to model distinct real-world memory-augmentation behaviors. The taxonomy spans object, conversational, visual, relational, temporal, and prospective memory operations, and participant reasoning supports distinctions beyond surface wording.

  • A Task Description Supplementary Information: The supplementary section expands the task taxonomy introduced in Section 3 and its comparison with prior datasets.
  • A.1 Task Taxonomy: The dataset categorizes question-answer pairs into six distinct functional tasks, each mimicking a real-world application of human memory augmentation.
  • A.1 Task Taxonomy: Object & Location Memory identifies an object’s last known position or tracks its movement across times and locations to maintain object permanence.It evaluates indexing the wearer’s data to locate an object based on when it was last observed.
  • A.1 Task Taxonomy: Conversational Memory recalls facts from multi-topic chats, including commitments, deferred answers, and mid-conversation corrections across shifting exchanges.The passage identifies this task with Dialogue State Tracking and emphasizes maintaining context without losing conversational awareness.
  • A.1 Task Taxonomy: Visual Scene Recall retrieves details from past environments, requiring recognition of objects, reading text, and linking visual elements to factual meaning.Examples include whiteboard text, manual instructions, and information displayed on a screen.
  • A.1 Task Taxonomy: In-Context Retrieval performs multi-hop reasoning by chaining disjoint facts from the user’s history and using a primary fact as prerequisite context.This task evaluates relational memory: representing and navigating associations between independent elements of an experience.
  • A.1 Task Taxonomy: Timeline Reconstruction sequences disjointed events chronologically across longer horizons, such as ordering locations visited during a multi-location errand.The task evaluates temporal episodic memory and can model procedural-memory tracking in structured activities.
  • A.1 Task Taxonomy: Intent Recall retrieves explicit reminders or passively recalls implied, incomplete goals, connecting prospective memory with future actions through time, location, or person-based triggers.Participant reasoning mapped representative questions to distinct memory operations, supporting a taxonomy that extends beyond surface wording.

B Data Collection Supplementary Information … C.2 Phase 2: Agentic QA Generation

SuperMemory-VQA was built from multimodal egocentric recordings collected under an IRB-approved protocol with privacy protections, then annotated through a two-phase, human-reviewed agentic pipeline. Phase 2 grounds generated question-answer pairs through ledger-based verification, iterative enhancement, rejection, and final review, with costs increasing as session duration grows.

  • B.1 Hardware and Modalities: Participants wore Meta Aria Glasses capturing RGB, SLAM, eye-tracking, audio, IMU, and orientation data at specified resolutions and sampling rates.The hardware collected 1408 × 1408 RGB video at 30 fps, two 640 × 480 grayscale SLAM streams at 30 fps, eye tracking at 60 fps, and seven audio channels at 48 kHz.
  • B.2 Protocol: 10 participants were recorded under an IRB-approved protocol evaluating multimodal, long-term memory retrieval for an AI-driven wearable system.Participants came from the general population and university settings.
  • B.2 Protocol: Sessions simulated home environments in short-term rentals, combined indoor and outdoor segments, and included guided exploration, glasses calibration, and object-location notes.Each session included at least one lab member and one to three external participants.
  • B.2 Protocol: The protocol classified the study as minimal risk and non-invasive, mitigating fatigue and overheating risks through breaks, device safeguards, and temperature monitoring.The protocol identified no direct participant benefit but described societal benefits for future AR memory-support systems.
  • B.3 Privacy and Anonymization Steps: Released data omit raw audio and instead provide manually redacted WhisperX transcripts, while RGB faces and license plates are blurred with EgoBlur.The release includes processed gaze, trajectory, SLAM point-cloud, and IMU data.
  • B.3 Privacy and Anonymization Steps: Privacy protection combines off-peak recording choices, removal of non-consenting indoor facial data, automated blurring, and manual verification of incidental captures.Automated tools and multimodal LLMs also flag potentially identifying text, followed by manual review before broad sharing.
  • C.1 Phase 1: Dense Video Captioning: The annotation pipeline uses two agentic phases initialized from chunked session videos, temporal metadata, WhisperX transcripts, and LLM-generated dense video captions, with human review in each phase.Chunking addresses LLM context limits, while participant-session audio is combined before transcription.
  • C.2 Phase 2: Agentic QA Generation: Phase 2 builds a unified Super Ledger, then uses QA Planner, Verifier, Retriever, and Enhancer agents to propose, ground, revise, reject, and human-review annotations.For T=1, generation costs approximately $0.026, three verification loops total approximately $0.752 per QA; for T=50, the corresponding total is approximately $1.241 per QA.

C.3 Pipeline Cost Analysis · D Verification Criteria and Annotation Format

The annotation pipeline’s cost is dominated by repeated multimodal verification and Pro QA generation rather than captioning, while conservative assumptions make projected costs sensitive to ledger narrowing, evidence duration, and pricing thresholds. SuperMemory-VQA validation combines agentic scoring, causal temporal filtering, schema checks, and multimodal annotation formats that support grounded abstention and long-term recall.

  • C.3 Pipeline Cost Analysis: 295 tokens per second is the combined media density for video with audio under Gemini API accounting, using 263 video and 32 audio tokens per second.Pricing uses the May 2026 Gemini Developer API standard paid tier, with gemini-3-flash-preview charging $0.50/M input tokens for text/image/video, $1.00/M for audio, and $3.00/M output token.
  • C.3 Pipeline Cost Analysis: The cost model splits each hour into thirty 120s chunks, with each chunk contributing 35.4k media tokens plus approximately 5k prompt/context tokens.Stage 2 adds Pro QA generation, text-only verification, Flash retrieval, Pro verification over 10 evidence clips totaling 300s, and Flash enhancement over the same evidence budget.
  • C.3 Pipeline Cost Analysis: $42 is the approximate cost of captioning 50 hours, compared with approximately $6.20k for Pro QA generation plus three-loop verification of 5,000 QAs.The default configuration uses Flash captioning and repeated multimodal evidence verification with Pro QA generation over a long Super Ledger.
  • C.3 Pipeline Cost Analysis: The 50 hour, 5,000 QA extrapolation is intentionally conservative because it sends a 15k-token/hour Super Ledger prefix during Pro QA generation and each Flash retrieval loop.Indexing or symbolic search can reduce Stage 2 text cost, whereas evidence bundles exceeding 300s or Pro requests crossing the 200k-token pricing threshold increase costs.
  • D.1 Verification Criteria: Candidate QA pairs receive continuous Verifier Agent scores for factual correctness, objective relevance, and causal answerability, and are rejected if any score falls below τ = 0.6.Objective relevance requires a temporal gap greater than 10 minutes and multi-clip reasoning; causal answerability requires resolution using evidence available by the execution timestamp.
  • D.1 Verification Criteria: Evidence starting after the question is programmatically pruned, and every MM:SS anchor must remain within its associated video segment.These deterministic checks enforce physical causality and prevent temporal hallucinations or drift.
  • D.1 Verification Criteria: Each evaluation question has four ordered choices: correct, vague, wrong, and N/A, where N/A supports abstention when available evidence is insufficient.Stored answerable pairs contain one correct, one vague, and one incorrect custom choice; unanswerable pairs contain three incorrect distractors, with explanations for each choice.
  • D.2 Detailed Input/Output Format: Questions and answers reference time spans, locations, and relevant sensor modalities, while evidence may span multiple sessions and always precedes the question start.The format records evidence descriptions, sessions, video IDs, timestamps, modalities, answerability, explanations, and task metadata for grounded long-term recall.

E Dataset Statistics Supplementary Information · F Human Review

SuperMemory-VQA combines long-horizon, variable-complexity evidence demands with a human review workflow that verifies grounding, correctness, causality, answerability, naturalness, and answer-choice balance. Most temporal gaps are 1–2 hours, though some require recalling events from over a week earlier, while timeline tasks require integrating multiple clips.

  • E Dataset Statistics Supplementary Information: Evidence complexity varies by task: object localization may need one segment, whereas timeline reconstruction requires clues from multiple separate clips.The number of distinct evidence clips is summarized in Figure 21b.
  • E Dataset Statistics Supplementary Information: Most questions have a 1–2 hour temporal gap, although extreme cases require recalling events recorded over a week earlier.The temporal gap measures elapsed time between the query and recorded evidence.
  • E Dataset Statistics Supplementary Information: Evidence duration provides a complementary difficulty measure, distinguishing questions answerable from brief moments from those requiring prolonged activity tracking or observation aggregation.Figure 22 reports total evidence duration and its distribution across task categories.
  • F Human Review: After Phase 1, reviewers inspect dense session captions in a platform showing processed video, caption files, local timeline windows, and interval-specific caption chunks.Reviewers can mark caption files accepted, rejected, or pending.
  • F Human Review: After Phase 2, reviewers compare generated question-answer annotations with source video and evidence timelines using temporal markers and annotation controls.The QA-review interface displays annotation files, video sources, evidence markers, and a side panel for the current annotation.
  • F Human Review: Reviewers can edit task labels, questions, reasoning, answer choices, evidence spans, modalities, bounding boxes, and human-review status, while inspecting raw model outputs during debugging.They can also navigate, verify scores, and add or delete annotations.
  • F Human Review: Human review is integrated into annotation, checking factual grounding, temporal causality, answerability, naturalness, and answer-choice balance before benchmark inclusion.Session attendants or context-familiar reviewers helped resolve ambiguous references and intended actions.

G Evaluation Protocol and Data Use

SuperMemory-VQA is evaluated in an open zero-shot setting without training on its QA labels, using only evidence available before each question. The released multimodal data support reproducible research, while current baselines use only subsets of the available sensor streams.

  • Evaluation protocol: Table 2 reports open zero-shot evaluation without training or fine-tuning on SuperMemory-VQA QA labels.Each system receives the question, answer choices, and evidence available before the question time.
  • Evaluation protocol: Released labels prioritize reproducibility and flexible research use over maintaining a hidden test split.The labels may also support future fine-tuning, according to the provided passage.
  • Baseline modality use: Current baselines do not exhaust the released RGB, transcript, gaze, motion, trajectory, IMU, and SLAM-derived data.Video-RAG mainly uses RGB frames, ASR transcripts, OCR, object detections, and retrieved auxiliary text; EgoButler uses RGB/audio-derived captions and hierarchical text memories.

H Reproducibility and Compute

The evaluation uses answerability F1, QA accuracy, and QA mean reciprocal rank. Reproducibility is supported through specified hardware and APIs, along with released evaluation scripts for baseline comparisons.

  • Evaluation metrics are answerability F1, QA accuracy, and QA mean reciprocal rank.
  • Open-source model evaluations ran on a server with 4×A100 GPUs.
  • Gemini-family models used the Google Cloud Platform API, while OpenAI-family models used Azure OpenAI APIs.
  • The released code repository contains evaluation scripts for reproducing the reported baseline comparisons.
Loading 2606.00825v1…