CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video

1 Introduction

• Continuous egocentric video processing is costly, visual-token budgets restrict frame coverage, and long multimodal contexts can worsen fine-grained perception and lost-in-the-middle failures.Whether captions support long-form egocentric episodic memory remains largely unexplored.Source: §1 · p. 1
• CapMem defines Episodic Memory Video Caption QA and introduces a human-annotated benchmark of 1,000 questions over 75 videos totaling 33.7 hours across 16 scenarios.Evidence timestamps and fine-grained visual tags support precise error analysis.Source: §1 · p. 1
• On videos longer than 20 minutes, full-coverage CaptionQA outperforms direct VideoQA for 10/12 models with 30-second captions and 8/12 with 60-second captions.Across all videos, the corresponding counts are 8/12 and 6/12; direct VideoQA remains competitive on shorter videos.Source: Figure 1 · p. 2
• CaptionQA gains persist after frame matching, with mean long-video gains of 3.22 and 2.55 points across six Qwen models.The 30-second setting has the clearest statistical evidence.Source: Figure 1 · p. 2
• The retrieve-and-verify harness uses captions as a semantic index and selectively revisits sparse visual evidence, improving most open-weight models.The paper presents caption memory as effective for long-form egocentric reasoning under current VLM constraints.Source: §1 · p. 2

2.1 Egocentric Datasets and Benchmarks

• Existing assistant-oriented datasets and QA benchmarks mainly cover short or minute-scale contexts, while CapMem provides a human-annotated long-form video benchmark.Some newer egocentric QA efforts use automatic or MLLM-generated questions, which may inherit generation-induced language bias.Source: §2.1 · p. 2

2.2 Episodic Memory Tasks

• Earlier episodic-memory QA work mainly studies visual retrieval or direct VideoQA, whereas this work evaluates caption-based memory for long-form egocentric QA.Related efforts include memory-oriented tasks in Ego4D and later egocentric episodic-memory QA and cross-view reasoning.Source: §2.2 · p. 2

2.3 Video Caption and Harness Engineering

• Caption-based reasoning in continuous, long-form egocentric settings remains underexplored despite prior work using clip- or frame-level captions for other video tasks.CapMem offers a dedicated testbed for studying how harness design can enhance caption-centric reasoning.Source: §2.3 · p. 3

3 Task and Evaluation Setup

• The evaluation framework uses three task settings, a two-stage pipeline, and a five-dimensional analysis of model capabilities.The settings include CleanQA, direct VideoQA, and CaptionQA.Source: §3 · p. 3, §3.1 · p. 3

3.1 Task Formulation

• Episodic Memory Video Caption QA asks models to perform multiple-choice episodic recall using only their self-generated timestamped captions.CaptionQA first converts temporally segmented video frames into captions; VideoQA answers directly from uniformly sampled frames.Source: §3.1 · p. 3

3.2 Benchmark Scope and Statistics

• CapMem contains 75 egocentric videos totaling 33.7 hours and 1,000 multiple-choice questions across 16 scenarios, with 62.7% of videos longer than 20 minutes.79.2% of questions are assigned to long videos, and each question has four options and an annotated evidence timestamp.Source: §3.2 · p. 3

3.3 Evaluation Framework and Dimensions

• Stage 1 gates models using CleanQA scores of 15%–35%; Stage 2 compares VideoQA and CaptionQA for models that pass.The analysis covers overall accuracy, computational cost, fine-grained visual accuracy, evidence-location accuracy, and duration-wise accuracy.Source: §3.3 · p. 4

4.1 Video Collection

• The video pool is drawn from seven egocentric datasets, and released materials comprise annotations and metadata rather than source videos or frames.The annotations and metadata are released for non-commercial research purposes, subject to dataset-specific licensing conditions.Source: §4.1 · p. 4

4.2 Question Generation

• Annotators use 50 templates organized by Ego/Exo subject and 4W1H intent, then instantiate them with video-grounded events and fine-grained visual cues.Answer options include distractors requiring discrimination of object identity, attributes, spatial relations, or actions.Source: §4.2 · p. 4

4.3 Question Filtering

• A four-model text-only blind test removes questions answered correctly by at least three models, and human validators discard queries with ambiguous or subjective visual evidence.A CleanQA gate further controls residual bias during evaluation.Source: §4.3 · pp. 4–5

5.1 Experiment Setting

• The evaluation uses single-turn, zero-shot inference and compares full-coverage captions from consecutive 30- or 60-second windows with direct video input under model-specific frame limits.A frame-aligned control uses six Qwen models and up to 768 frames; cross-captioning pairs six Qwen captioners with two Qwen QA models.Source: §5.1 · p. 5, §5.2 · p. 5

5.2 Main Results

• With each model’s better caption window, full-coverage CaptionQA outperforms VideoQA for 8 of 12 models, but VideoQA remains stronger for both InternVL3.5 models and Gemini 3 Flash.For example, Qwen3-VL-32B rises from 42.8% to 47.7% with 30-second CaptionQA; 30-second windows generally outperform 60-second windows.Source: §5.2 · p. 5, Table 3 · p. 5
• On videos longer than 20 minutes, CaptionQA–VideoQA gaps turn positive for most models, reaching +7.2 points for Qwen3-VL-32B with 30-second captions and +15.2 points for GPT-5.2 with 60-second captions.On videos of at most 20 minutes, CaptionQA generally trails direct VideoQA.Source: Figure 3 · p. 6
• After frame alignment, CaptionQA’s mean gains over VideoQA are +3.22 and +2.55 points on long videos at 30 and 60 seconds, respectively.The 30-second confidence interval is positive on long videos, [+0.45, +6.07], while the 60-second interval is negative on short videos, [−9.17, −1.29].Source: §5.2 · p. 6, Table 5 · p. 6, Table 4 · p. 6
• With the QA model fixed, captioner choice produces a 2.2–6.1-point max–min accuracy spread, and Qwen3.5-2B scores 39.8% with Qwen3-VL-32B’s captions versus 36.2% with its own.CaptionQA performance depends on both memory construction and downstream reasoning.Source: Figure 4 · p. 6, §5.2 · p. 6, Table 6 · p. 7
• Caption memory amortizes visual processing: Qwen CaptionQA falls from 330k–335k to 14k–20k tokens per question when reuse increases from 1Q/video to 32Q/video.Constructing both caption granularities uses 101,814 frame inputs versus 725,973 for Direct VideoQA, a 7.1× reduction in repeated visual-frame processing.Source: §5.2 · p. 6
• For several models, full-coverage CaptionQA produces a flatter three-bin evidence-position accuracy profile; for GPT-5.2, it largely removes the observed middle-position drop.The pattern is not universal and does not establish a causal mechanism.Source: Table 7 · p. 7

5.3 Harness Engineering

• The retrieve-and-verify harness improves all open-weight models in both caption settings, with Qwen3.5-2B gains of +4.8 at 30 seconds and +5.3 at 60 seconds.It retrieves caption windows, checks sparse frames, and overrides the baseline answer only when verification returns a strongly supported valid option without an explicit clock-time cue.Source: §5.3 · p. 7, Figure 6 · p. 7

• CapMem is a human-annotated benchmark for caption-based egocentric episodic memory.Source: §6 · p. 7
• Across 12 VLMs, CaptionQA gains concentrate on long videos, and the mean gain remains positive under matched frames across six selected Qwen models.Source: §6 · p. 7
• Retrieve-and-verify improves most open-weight models, supporting caption memory as an effective approach under current VLM constraints.Source: §6 · p. 7

• CapMem contains 75 public egocentric videos and 1,000 QA pairs, covering only a subset of real-world wearable scenarios.Source: Limitations · p. 8
• Ego4D contributes 56 of 75 videos and 742 of 1,000 questions, so aggregate results may partly reflect its capture distribution.Source: Limitations · p. 8
• The primary visual comparison is standard Direct VideoQA under supported frame budgets; frame-aligned and cross-captioning controls are restricted to Qwen models.Source: Limitations · p. 8

Ethical Considerations

• CapMem’s annotations cover observable events, objects, actions, and spatiotemporal evidence, but avoid inferred identities, sensitive attributes, and personal intentions; source videos are not redistributed.The release provides annotations and metadata for non-commercial research, subject to source-dataset restrictions; deployment also calls for consent, data minimization, and access control.Source: Ethical Considerations · p. 8

A An Introduction to Episodic Memory

• Episodic recall in egocentric video requires structured encoding, cue-driven retrieval, and evidence-grounded answers to questions about a wearer’s past experience.Captions serve as searchable retrieval cues, but their lossy descriptions can omit details that require checking raw visual evidence.Source: A An Introduction to Episodic Memory · pp. 12–13

B Guiding Taxonomy and Question Template for QA Generation

• The annotation taxonomy combines Ego/Exo perspective with 4W1H intent, while 50 templates guide broad, consistent, and diagnostically useful question coverage.The templates scaffold natural questions about concrete events without rigidly prescribing wording; downstream filtering removes shortcuts and ambiguous cases.Source: B Guiding Taxonomy and Question… · pp. 13–14

C Prompts

• Task-specific prompts are shared across evaluated models for each input setting; the 30s and 60s CaptionQA settings differ in window length and resulting captions, not prompt template.The prompt instructions are shown in Figure 7.Source: C Prompts · p. 14

Harness policy

• The harness keeps the CaptionQA baseline as the default and overrides it only when visual verification strongly supports a valid answer and no explicit clock-time cue is present.Clock-time questions are excluded because sparse frames may miss the referenced moment.Source: D Harness Engineering · pp. 14, 17, Figure 8 · p. 17

Caption-window retrieval

• Caption windows are ranked using the question and answer options with four retrieval signals: BM25, hash-cosine similarity, word overlap, and time hints.The signals address exact keyword matches, phrasing variation, broad query coverage, and temporal cues, respectively.Source: D Harness Engineering · pp. 15–16
• Retrieval selects up to three seed windows, suppresses seeds within 45 seconds of an existing one, and adds one neighboring window on either side when available.This yields at most nine caption windows; the retrieved-caption answer is an intermediate candidate for visual verification, not the final prediction.Source: D Harness Engineering · p. 16

Sparse-frame verification

• Visual verification uses at most three intervals and four sampled frames per interval, for a maximum of 12 frames per question.Intervals are centered on retrieved seed windows and expanded with neighboring windows and five seconds on each side when available.Source: D Harness Engineering · p. 17

Conservative caption-primary policy

• The final prediction remains the baseline unless visual support is strong, the visual answer is a listed option, and the question/options lack an explicit clock-time cue.This conservative rule blocks overrides for clock-time questions because sparse frames may miss the referenced moment.Source: D Harness Engineering · p. 17

Evaluation caveat

• The override policy was chosen after inspecting errors on the full benchmark, so the reported gains are a post-hoc proof of concept rather than an unbiased out-of-sample estimate.CapMem has one 1,000-question evaluation split and no separate development set.Source: D Harness Engineering · p. 17

E Annotation and Quality-Control Pipeline

• Twelve trained annotators created 2,109 candidate QA pairs, and two reviewers checked solvability, answers, templates, and category assignments before retaining 1,276.Reviewers resolved disagreements by consensus.Source: E Annotation and Quality-Control… · p. 18
• A text-only ensemble flagged 352 candidates as answerable without video; reviewers restored 76 false positives and verified the final 1,000 questions as unambiguously answerable from source video.The automated filtering left 924 candidates before the manual re-examination.Source: E Annotation and Quality-Control… · p. 18, Table 9 · p. 19
• Multi-label visual tags classify queried information as Object, Attribute, Spatial, or Action, with definitions checked during review.The categories cover visible entities, observable properties, spatial relations, and actions or state transitions.Source: E Annotation and Quality-Control… · p. 18
• The benchmark spans 33.7 hours, 16 scenarios, and four application domains, but is intended as a quality-controlled evaluation set rather than a comprehensive sample of wearable-video settings.The authors attribute its scale to the cost of manual construction, review, shortcut filtering, and visual-solvability verification.Source: E Annotation and Quality-Control… · p. 18

F Supplementary Experimental Settings and Results

• The supplementary comparison uses full-coverage CaptionQA with consecutive 1-FPS frames grouped into 30- or 60-second windows, alongside direct VideoQA and a frame-aligned control using the same selected frames.CaptionQA reuses timestamped captions across a video's questions without accessing frames during downstream QA.Source: §F.1 · p. 18, F Supplementary Experimental Settings… · p. 18

F.1 Supplementary Experimental Settings

• Supplementary settings document open-weight hardware and software configurations, closed-source request parameters, and decoding and response-scoring policies for reproducibility.The reported configurations include vLLM and Transformers versions for open-weight inference and OpenRouter settings for closed-source models.Source: §F.1 · pp. 18–19

F.4 Duration Regimes and the VideoQA Frame Cap

• 3.22 and 2.55 points: frame-aligned CaptionQA exceeds Direct VideoQA on videos over 20 minutes at 30- and 60-second intervals; crossing the 768-frame cap alone is insufficient.The intermediate >12–≤20-minute group has a zero aligned gap at 30 seconds and a negative gap at 60 seconds; the >20-minute threshold is an empirical CapMem regime, not a universal or cap-invariant threshold.Source: §F.4 · pp. 20–21, Table 12 · p. 21

F.5 Uncertainty Estimation

• Uncertainty is estimated by duration-stratified, video-cluster bootstrap, retaining all questions and six Qwen results per sampled video; intervals cover videos, not model families.The 95% intervals use 10,000 bootstrap samples.Source: §F.5 · p. 21

F.6 Frame Accounting, Runtime, and Cost

• CaptionQA builds memory once per video, while Direct VideoQA reprocesses frames per question; costs vary by model, and memory reuse amortizes as more queries share it.Gemini’s timing run is separate from the main evaluation, and its recorded API charge is a lower bound.Source: §F.6 · pp. 21–22

F.3 Full Evidence-Location Analysis

• Several models show middle-position drops under Direct VideoQA; GPT-5.2’s CaptionQA profile is flatter, though flatter profiles need not mean higher accuracy.The pattern is not universal, and the 30-second setting is flatter than the 60-second setting for several models.Source: §F.3 · p. 20, Figure 9 · p. 23

F.2 Fine-Grained Visual Detail Results

• GPT-5.2’s 30-second CaptionQA beats Direct VideoQA across all four fine-grained tags, while Gemini 3 Flash is stronger with Direct VideoQA across all four.Across open-weight models, differences vary by architecture and tag; caption memory may omit small objects, subtle attributes, spatial relations, or brief actions.Source: §F.2 · pp. 19–20, Figure 10 · p. 24

G Licensing and Release Details

• Appendix G distinguishes CapMem-authored artifacts from third-party source media and describes their respective release terms.Source: G Licensing and Release Details · p. 24

G.1 Release Composition and Source-Dataset Licensing

• CapMem releases 1,000 multiple-choice QA records with answer keys, evidence timestamps, labels, source identifiers, an evaluation split, prompts, and scripts.The records represent 75 videos totaling 33.7 hours.Source: §G.1 · p. 24
• The release excludes source videos, frames, audio, and copied source annotations; users must obtain media from providers and follow their terms.Source: §G.1 · p. 24

G.2 CapMem License

• CapMem annotations and metadata use CC BY-NC 4.0, while evaluation and retrieval scripts use Apache 2.0; source media remains governed by original terms.Source: §G.2 · p. 24