PaperlayerView exact revision on arXiv

Source-linked AI summary

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants

Jianguo Huang, Jinming Liu, Qiyao Wang, Liang Xu, Jianhang Li, Zhimian Wen, Mingda Li, Shule Lu, Zhicheng Wang, Yuhan Guo, Xin Jin, Wenjun Zeng

arXiv:2609.37559v1cs.CV

TL;DR

Streaming assistants need memory across real-world interactions separated by interruptions, but existing benchmarks and methods largely focus on individual continuous videos. APM-Bench evaluates persistent memory across multi-session life trajectories and finds that existing methods face a utility–latency–storage trade-off, including difficulty acknowledging unavailable evidence.

  • Problem

    Most streaming benchmarks and methods focus on individual continuous videos, leaving persistent memory across intermittent interactions insufficiently evaluated despite its role in retaining and reusing user-specific experience.

    Source: §1 · p. 1

  • Method

    APM-Bench organizes egocentric experience into activity-related multi-session trajectories and evaluates memory across cross-session understanding, real-time perception, and adaptive response tasks.

    Source: §1 · pp. 2–3

  • Results

    Existing methods face a utility–latency–storage trade-off, and even the strongest general video model struggles to acknowledge insufficient historical evidence.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    APM-Bench provides a testbed for comparing persistent-memory systems under realistic streaming conditions while jointly considering utility, latency, and storage.

    Source: §5 · p. 10

Abstract

from arXiv · show

To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use. Yet existing streaming benchmarks and methods often focus on individual continuous videos or short clips, overlooking that real-world interactions are often intermittent and require memory to persist across interruptions. To fill this gap, we introduce APM-Bench, which reformulates real-world streaming interaction as multi-session life trajectories. It contains 549 sessions, 104 trajectories, and 2,719 candidates, spanning both objective and open-ended questions. Each session is a video with fine-grained annotations, and sessions within a trajectory revolve around related activities. Models then use persistent memory to answer questions about past sessions and provide proactive responses while maintaining real-time interaction. This raises challenges: persistent memory must be storable, selectively retain information, be injected at the right time, and remain efficient. Moreover, finite storage may leave required evidence unavailable, so assistants should recognize missing evidence. Therefore, we systematically evaluate general video models under different memory protocols and diverse specialized streaming memory systems, and test whether models acknowledge insufficient evidence. Our evaluation reveals a clear utility--latency--storage trade-off: existing methods still struggle to simultaneously achieve reliable long-term recall, low overhead, and effective proactive assistance across sessions. APM-Bench provides a comprehensive testbed for developing and comparing persistent memory systems under realistic streaming conditions. We hope it encourages future work that jointly considers utility, latency, and storage toward more practical persistent memory for real-world streaming assistants.

1 INTRODUCTION

Existing streaming benchmarks and methods largely focus on individual continuous videos, while real-world interactions are intermittent and require memory across interruptions. APM-Bench evaluates persistent memory across related multi-session trajectories, including its utility, deployment costs, and response to unavailable evidence.

1 INTRODUCTION

  • APM-Bench organizes egocentric experience into activity-related multi-session trajectories and evaluates cross-session understanding, real-time perception, and adaptive response through 12 tasks.Sessions preserve temporal order and gaps and include fine-grained annotations; the benchmark also tests responses when historical evidence is unavailable.Source: §1 · pp. 2–3
  • The benchmark addresses persistent-memory choices about what to store, when to use it, and how to keep storage and latency manageable.Finite storage can leave relevant visual history unavailable, so assistants should acknowledge missing evidence rather than fabricate an answer.Source: §1 · pp. 2–3
  • Video-as-memory preserves visual details but costs more storage and latency and often weakens real-time perception; specialized systems vary in efficiency, yet most provide weak service quality.Even the strongest general video model struggles to acknowledge insufficient historical evidence in controlled tests.Source: §1 · p. 3
  • APM-Bench systematically analyzes memory representations and systems across task performance, storage, and latency, revealing existing systems’ strengths and limitations..Source: §1 · p. 3
  • The paper aims to encourage persistent-memory research that jointly considers utility, latency, and storage for more usable real-world assistants.Source: §1 · p. 3

2 RELATED WORK

Prior streaming benchmarks and memory systems cover online understanding, interaction, recall, proactive assistance, and diverse ways to represent or retrieve history. However, most do not address persistent memory across interrupted sessions together with its storage and latency costs.

Streaming Video Benchmarks

  • Figure 3 characterizes six streaming formulations across 12 tasks, distinguishing intra-session from inter-session evidence and prompted from autonomous adaptive responses.Cross-session understanding uses prior-session evidence; real-time perception uses current-session evidence, while adaptive-response tasks use either.Source: Figure 3 · p. 4
  • Most streaming benchmarks evaluate single continuous videos, while APM-Bench focuses on whether memory remains useful across temporally separated sessions and at what storage and latency cost.Prior work also includes long-horizon episodic recall and proactive service, alongside benchmarks for response timing and content.Source: §2 · pp. 3–4

Streaming Memory Systems

  • Streaming memory systems use visual, event- or object-structured, KV-cache, text-based, and parametric representations, but most manage context within one continuous video.Persistent-memory systems exist, yet retrieval and reasoning can introduce substantial response latency, and questions remain about storage affordability and unavailable evidence.Source: §2 · p. 4

3 APM-BENCH

APM-Bench organizes egocentric video into activity-related, multi-session trajectories and evaluates persistent memory across historical recall, current-scene perception, and adaptive response. Its construction and evaluation protocols capture intermittent timelines, unavailable evidence, and memory costs.

3 APM-BENCH

  • APM-Bench evaluates cross-session understanding, real-time perception, and adaptive response across 12 task types, using multiple-choice questions for the first two families and open-ended questions for adaptive response.Tasks are grouped as intra-session or inter-session according to evidence location.Source: §3 · pp. 4–5

3.1 BENCHMARK CONSTRUCTION

  • The benchmark organizes EgoLife and HD-EPIC videos into activity-related, timestamped trajectories, then filters and refines generated candidates through automated and human review.The pipeline yields 2,719 candidates; 300 sampled questions received Cohen’s kappa of 0.868 between two annotators.Source: §3.1 · p. 5, Figure 5 · p. 6
  • Its evidence-availability set tests whether models answer when evidence is present and acknowledge its absence, using 260 questions and access to only the two most recent completed sessions.Half the questions have required evidence within that history; half require evidence from earlier sessions.Source: §3.1 · p. 5

3.2 DETAIL OF APM-BENCH

  • Adaptive Response covers five tasks, including waiting for sufficient evidence, checking registered conditions, and bridging past experience with current activity for guidance or reminders.Evidence-Ready Answering and Registered-Condition Response are intra-session tasks; the other three bridge sessions.Source: §3 · p. 5, §3.2 · p. 6

3.3 EVALUATION PROTOCOL AND METRICS

  • At inference, models receive prior-session memory and the current session’s causal video prefix; Adaptive Response is evaluated at timestamped probes, while the other two families use query-time instances.The Gated LLM-Judge Score checks intervention decisions and, for positive Evidence-Ready Answering probes, the selected option; passing responses receive a rationale score, scaled to a 0–100 task score.Source: §3.3 · pp. 6–7
  • Storage cost measures the peak persistent state retained across completed prior sessions, excluding the ongoing session, and reports it per hour.Trajectories average 69 minutes of video, but their real-world spans extend across hours or days because of inter-session gaps.Source: §3.2 · p. 6, §3.3 · p. 7

4 EXPERIMENT

The experiments compare general video models and specialized memory systems across cross-session understanding, real-time perception, adaptive response, and deployment costs. Results expose trade-offs in memory utility, latency, and storage, while showing persistent difficulty with selective memory use and proactive assistance.

4.1 EXPERIMENT SETUP

  • General video models use no-memory, raw-video, or session-summary conditions, while eight specialized methods span five memory representations.All models use 1 FPS input; specialized methods process prior sessions sequentially.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Raw video achieves the strongest cross-session performance for most general models but incurs substantial storage and latency costs; event-structured methods lead among specialized systems.Across methods, stronger utility, low latency, and low storage rarely coincide.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • Replacing raw-video history with text summaries improves real-time perception for most general models, supporting selective injection of history when relevant.The authors note that excessive or irrelevant history can interfere with current-scene understanding.Source: §4.2 · p. 9
  • Strong recall does not ensure recognition of unavailable evidence: Gemini 3.6 Flash performs strongly when evidence is accessible but is less reliable when it is unavailable.SimpleStream shows the opposite pattern, favoring the insufficient-evidence option but rarely answering correctly when historical evidence is available.Source: §4.2 · p. 9

4.3 WHY ADAPTIVE RESPONSE REMAINS DIFFICULT

  • Models perform substantially better on registered reminders than on autonomous assistance tasks, where no explicit trigger identifies relevant history.Even with raw-video memory or only necessary historical evidence, models remain weak at connecting past experience to the current scene and providing useful assistance.Source: Table 4 · p. 9, §4.3 · p. 10

5 CONCLUSION

APM-Bench organizes egocentric experience into activity-related multi-session trajectories to evaluate persistent memory under intermittent use. The conclusion finds that current methods struggle to balance recall, efficiency, selective memory use, and proactive assistance.

5 CONCLUSION

  • APM-Bench organizes egocentric experience into activity-related multi-session trajectories to simulate intermittent use and evaluate streaming-video persistent memory.The benchmark’s sessions preserve activity relations and temporal order across interruptions.Source: §5 · p. 10
  • Raw video preserves the richest visual evidence and gives the strongest cross-session performance, but carries substantial storage and latency costs.Compact representations reduce costs but may lose details; event-structured memory has the strongest utility among specialized systems.Source: §5 · p. 10
  • Current methods still struggle to combine reliable long-term recall, selective memory use, low overhead, and effective proactive assistance across sessions.Strong recall does not guarantee awareness when required evidence is unavailable, and adaptive response remains difficult even with rich history.Source: §5 · p. 10

A.1 COMPUTE

The appendix specifies shared compute conditions, evaluated backbones, and procedures for judging adaptive responses. The judge combines deterministic decision checks with rubric-based assessment of response quality.

A.1 COMPUTE

  • All experiments use H20 GPUs with the same host configuration to compare latency and storage costs across open-source video models and specialized memory systems.The shared host configuration is intended to enable fair comparison of those costs.Source: §A.1 · p. 15

A.2 BACKBONES

  • Table 5 lists the visual-language backbones used by specialized memory systems and SimpleStream.The supplied passage identifies the table’s scope but gives no backbone-level entries.Source: Table 5 · p. 15, §A.2 · p. 15

A.3 LLM-AS-JUDGE RUBRIC

  • After the deterministic gate, DeepSeek V4 Flash judges response quality using task guidance, the causal cutoff, reference annotations, and the model response.The evaluation assesses semantic quality rather than requiring lexical overlap or identical wording.Source: §A.3 · p. 15, DeepSeek V4 Flash: complete judge… · p. 15

DeepSeek V4 Flash: complete judge system rubric and output contract

  • The rubric scores responses from 1 to 5, from substantially inconsistent or unusable to task-faithful, grounded, accurate, precise, and useful.Scores of 4 and 3 allow minor imprecision or generic, incomplete responses, respectively.Source: DeepSeek V4 Flash: complete judge… · pp. 15–16

DeepSeek V4 Flash: task-specific guidance

  • Task-specific guidance checks whether history and the current scene support the relevant decision and response, including task continuity for TPG and registration fidelity for RCR and PRM.MPA guidance checks whether earlier experience improves assistance and whether the response is actionable.Source: DeepSeek V4 Flash: task-specific… · p. 16

B ADDITIONAL RESULTS AND ANALYSIS

The appendix defines additional measures and protocols for adaptive decision accuracy, response quality, latency, and persistent storage. It also reports unrecoverable output-format failures and notes that long histories increase such failures for some methods.

B.1 ADAPTIVE DECISION ACCURACY

  • Adaptive decision balanced accuracy averages correct INTERVENE rates on positive probes and correct SILENT rates on negative probes; the overall measure pools probes across five tasks before averaging polarity rates.This measure tests decision timing, while the main Gated Judge score also evaluates the response.Source: §B.1 · pp. 16–17, Table 6 · p. 17

B.2 RESPONSE QUALITY AFTER CORRECT DECISIONS

  • Response quality is judged after the gate, with available polarities weighted equally within each candidate before averaging candidates.The judge’s 1–5 score is converted to a 0–100 scale; Table 7 reports these scores separately from the main Gated Judge result.Source: §B.2 · pp. 17–18, Table 7 · p. 18

B.3 LATENCY

  • For local runs, query-to-first-token time includes memory preparation and visual processing; offline replay real-time factor divides prefix-processing time by video duration, excluding decoding.Proprietary API latency is measured end-to-end through the complete response.Source: §B.3 · p. 19, Table 8 · p. 19, Table 9 · p. 19

B.4 PERSISTENT STORAGE

  • Video-as-memory models have mean and maximum trajectory-peak storage of 2,316.01 and 9,639.01 MiB, respectively; text-summary memory occupies only kilobytes.Video-SALMONN S also maintains fast weights averaging 8.02 MiB at trajectory peak; these weights do not grow with processed video length.Source: §B.4 · p. 19, Table 10 · p. 19

B.5 INSTRUCTION FOLLOWING

  • Some memory methods show more unrecoverable output-format failures with long input histories, indicating reduced instruction following under heavy context.The reported failures exclude formatting errors the scorer can repair deterministically.Source: §B.5 · p. 20, Table 11 · p. 20, Table 12 · p. 20

B.6 EFFECT OF TRAJECTORY ORGANIZATION

The controlled trajectory comparison makes model differences more visible than Raw Lifelong on several capabilities, while oracle access and restricted-history tests show that evidence selection and unavailable evidence remain challenges.

B.6 EFFECT OF TRAJECTORY ORGANIZATION

  • APM-Bench produces a wider score spread than Raw Lifelong for Cross-session Understanding (8.75 vs. 3.33), Adaptive Response (8.03 vs. 4.84), and the mean of three capability scores (8.77 vs. 7.05).All three systems score higher on Cross-session Understanding with APM-Bench than with Raw Lifelong: 56.90 vs. 41.95, 40.05 vs. 34.16, and 37.02 vs. 35.97.Source: §B.6 · p. 20
  • Oracle raises Cross-session Understanding by 19.78–31.21 points over APM-Bench, while FluxMem’s Real-time Perception falls from 33.95 with Oracle to 25.14 with APM-Bench and 15.28 with Raw Lifelong.Adaptive Response gains less under Oracle and remains at 28.98–51.86, so access to past evidence alone does not ensure an appropriate response to the current scene.Source: §B.6 · p. 20, Figure 6 · p. 21

B.7 EVIDENCE AVAILABILITY AND ANSWER ACCURACY

  • On the 130 questions whose evidence remains in the two most recent sessions, Seed-2.0-Lite and Gemini 3.6 Flash score 64.62% and 85.38%, versus 62.69% and 76.15% originally.All eight specialized systems score below their original accuracy on this subset, including HERMES (7.69% vs. 38.46%); accessible evidence alone does not ensure a correct answer.Source: §B.7 · p. 21, Table 13 · p. 21

B.8 TASK-LEVEL PERFORMANCE

Figures 7 and 8 present scores across 12 tasks for general video models using video- and text-memory protocols and specialized streaming-memory methods, with SimpleStream as a comparison.

B.8 TASK-LEVEL PERFORMANCE

  • Both figures use a common 0–100 scale, enabling task-level score comparisons across their panels.The axes group tasks into Cross-session Understanding, Real-time Perception, and Adaptive Response.Source: §B.8 · p. 22, Figure 7 · p. 22, Figure 8 · p. 23
  • Figure 7 compares six general video models under video- and text-memory protocols with SimpleStream across 12 tasks on a 0–100 scale.The task groups cover Cross-session Understanding (ER–TR), Real-time Perception (ACR–STU), and Adaptive Response (ERA–TPG).Source: Figure 7 · p. 22, §B.8 · p. 22
  • Figure 8 compares eight specialized streaming-memory methods with SimpleStream across the same 12 tasks on a 0–100 scale.The task groups cover Cross-session Understanding (ER–TR), Real-time Perception (ACR–STU), and Adaptive Response (ERA–TPG).Source: §B.8 · p. 22, Figure 8 · p. 23

C DATASET STATISTICS

The dataset statistics describe trajectory activities, evidence composition and distance, and the contrast between retained video duration and elapsed real-world span.

C.1 TRAJECTORY ACTIVITIES

  • EgoLife trajectory titles and HD-EPIC recipe names supply the activity terms, with each term counted at most once per trajectory.Figure 9 sizes terms by the number of trajectories in which they occur.Source: §C.1 · p. 24, Figure 9 · p. 24

C.2 EVIDENCE COMPOSITION

  • Table 14 distinguishes candidates requiring at least two evidence items from those whose evidence spans at least two sessions.Rates use each row’s candidate count, and cross-day evidence means at least one decisive item precedes the query day.Source: Table 14 · p. 24, §C.2 · p. 24

C.3 EVIDENCE DISTANCE MEASURED IN SESSIONS

  • Among 1,249 inter-session candidates, decisive evidence occupies one, two, three, four, or five historical sessions for 987, 186, 65, 10, and 1 candidates, respectively.Evidence distance counts session boundaries to the earliest decisive evidence; PRM uses its earlier registration as the evidence anchor.Source: §C.3 · p. 24, Table 15 · p. 25

C.4 REAL-WORLD SPAN AND ORGANIZED TRAJECTORY DURATION

  • EgoLife retains 1.3 hours of video across a mean 79.9-hour span, while HD-EPIC retains 0.7 hours and overall retains 1.1 hours across 58.9 hours.The span includes gaps from first session start to last session end, whereas organized duration sums retained session videos; HD-EPIC’s structured recipes require less reduction than EgoLife.Source: §C.4 · p. 25, Figure 10 · p. 25

D.1 DATASET CONSTRUCTION

The appendix details how APM-Bench candidates are constructed, verified, and represented, then specifies task prompts and a query-agnostic text-memory protocol for streaming evaluation.

D.1 DATASET CONSTRUCTION

  • Stage 1 organizes raw videos into activity-related trajectories, retaining 76 EgoLife trajectories with 472 sessions and 28 HD-EPIC trajectories with 77 sessions after human verification.For EgoLife, GPT-5-mini generates hourly and daily summaries and GPT-5.4 mines trajectory proposals; HD-EPIC proposals derive from recipe-stage annotations.Source: §D.1 · p. 25
  • Stage 2 generates timestamped visual captions and 6,412 initial candidates across Cross-session Understanding, Adaptive Response, and Real-time Perception, followed by automated screening and human refinement.An independent audit of 300 final questions achieved Cohen’s kappa of 0.868.Source: §D.1 · pp. 25–26

D.2 HUMAN VERIFICATION

  • Human reviewers inspect source video and time targets alongside criteria and editable annotations, checking evidence support, timestamps, answerability, causal validity, and response or silence labels.Adaptive Response review also checks reference responses, ideal intervals, historical links or registrations, and whether interventions are necessary.Source: §D.2 · p. 26

D.3 MCQA ANNOTATION FORMAT

  • The seven MCQA tasks share a four-option question and evidence structure, with evidence moments specifying session, time interval, and content.STU uses the answer format with an additional instruction to interpret spatial relations from the camera wearer’s viewpoint.Source: §D.3 · p. 26, MCQA: ER, EST, TR, ACR, CT, OCR, STU · p. 26

D.4 ERA: EVIDENCE READINESS

  • ERA registers a multiple-choice question at session start and marks when evidence makes its answer available using positive and negative heartbeat timestamps.The assistant must remain silent until evidence is sufficient, then intervene with the selected option and rationale.Source: §D.4 · p. 26, ERA: question, evidence, and probe… · p. 27

D.5 RCR: IN-SESSION CONDITIONAL REMINDER; D.7 PRM: CROSS-SESSION REGISTERED REMINDER

  • RCR tests an in-session registered reminder, while PRM pairs an earlier registration with later probes for intervention or silence.Positive and negative probes record when to fulfill the reminder or remain silent; probe timestamps use heartbeat fields for RCR and interval end times for PRM.Source: §D.5 · p. 28, RCR: registration and probe labels · p. 28, §D.7 · p. 28

D.6 MPA: ASSISTANCE FROM PRIOR EXPERIENCE; D.8 TPG: CONTINUING-TASK GUIDANCE

  • MPA and TPG use historical evidence and current-scene probes to assess proactive guidance, recording reference responses and windows for intervention or silence.TPG concerns guidance for a continuing task; both task formats include reasons supporting response decisions.Source: §D.6 · p. 28, §D.8 · p. 29

E TASK PROMPTS; E.4 EVIDENCE AVAILABILITY-AWARE PROMPT; E.5 TEXT-SUMMARY MEMORY PROMPT

  • The shared system prompt restricts the assistant to the causal stream, while the text-memory protocol writes a detailed, query-agnostic session summary for later use without access to future questions.The evidence-availability prompt adds a fifth option for identifying questions whose required evidence lies outside the two most recent completed sessions.Source: §E.1 · p. 29, Shared system instruction · p. 29, §E.4 · p. 32

Related papers

Loading 2609.37559v1…