PaperlayerView exact revision on arXiv

Source-linked AI summary

GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience

Sitong Gong, Caixin Kang, Tianyu Yan, Guo Chen, Bo Zheng, Kaipeng Zhang, Yunzhi Zhuge, Xiang Ruan, Huchuan Lu, Yifei Huang

arXiv:2608.02392v2cs.CVcs.AI

TL;DR

Wearable assistants need one memory that supports both answering questions about visual history and proactively using past experience when relevant. GROVE causally organizes streaming video into temporal strata with scale-native retrieval, improving long-horizon recall and situation-aware assistance across five benchmarks.

  • Problem

    Existing systems largely separate question-conditioned recall from proactive assistance, lacking a shared streaming memory accessible through either user questions or unfolding situations.

  • Method

    GROVE causally organizes streaming video into perceptual traces, moments, episodes, and recurring patterns, pairing each temporal stratum with a specialized retrieval skill.

  • Results

    Across five benchmarks, GROVE’s temporal structure and scale-native retrieval improve both long-horizon recall and situation-aware assistance, with ablations supporting both components.

  • Takeaways & Limitations

    GROVE provides a shared memory substrate for assistants that continuously observe, remember, answer questions, and offer proactive help.

  • Takeaways & Limitations

    GROVE inherits errors from its frozen perception model, cannot recover evidence missed during ingestion, and becomes more costly as observed history grows.

Abstract

from arXiv · show

A wearable assistant should both answer questions about its visual history and recognize when that history is useful to the present situation. Existing video-memory systems primarily support question-conditioned recall, whereas proactive assistants typically use separate memory and control mechanisms. We introduce GROVE, a training-free framework that supports both behaviors with one memory grown causally from a continuous video stream. GROVE retains fine-grained perceptual evidence and incrementally consolidates it into time-stamped moments, coherent episodes, and recurring cross-day patterns. Each stratum is paired with a scale-native retrieval skill for locating an observation, replaying an activity, or traversing long-range regularities. Reactive QA and proactive assistance share this memory and access interface, differing in whether retrieval is initiated by a user query or the current situation. Across multiple benchmarks including the challenging MM-lifelong and EgoServe, GROVE achieves the best results among the compared methods. Controlled ablations show that the temporal strata and their access skills are complementary, with patterns providing the largest benefit when evidence spans multiple days. Code will be available at https://github.com/SitongGong/GROVE.

Introduction

GROVE addresses the need for wearable assistants to support both reactive recall and proactive intervention through one causally grown, temporally stratified streaming memory. Its scale-native retrieval interface organizes evidence from fine-grained observations into moments, episodes, and recurring patterns, achieving the best overall results among compared methods across five benchmarks.

  • Motivation: Wearable assistants must retrieve past evidence for user questions and proactively intervene when the current situation warrants assistance.These are complementary assistance modes: reactive question answering and proactive assistance.
  • Problem and approach: GROVE provides a single causal streaming memory accessible from either a past-evidence question or the current situation.This shared interface replaces separate query-first memory and intervention-oriented control mechanisms.
  • Temporal organization: Temporal scale links stored memory to retrieval: moments capture short events, episodes capture activities, and patterns capture regularities recurring across days.Long-range regularities are explicitly constructed because they are absent from individual frames or events.
  • Framework: GROVE is training-free and grows temporally stratified memory online from narrative captions and structured perceptual registries, then consolidates observations into moments, episodes, and patterns.The registry includes entities, attributes, explicit counts, actions, and on-screen text.
  • Evaluation: 5 benchmarks support GROVE’s effectiveness, with best overall results among compared methods and ablations validating its temporal memory organization and scale-aware retrieval.The evaluation covers lifelong question answering, online video understanding, forward assistance, and query-free proactive services.

Related Works

Prior streaming-video systems process incoming windows and maintain compact present-focused state, while memory-based approaches provide complementary training-free retrieval capabilities. GROVE instead retains structured, queryable memory across days and is evaluated on benchmarks covering real-time perception, tracing, active response, and contextual understanding.

  • Streaming Video Understanding: Offline long-video models extend context or compress clips before answering, an assumption that fails for continuously arriving video.The passage contrasts offline processing with streaming settings where video is still arriving.
  • Streaming Video Understanding: Online video LLMs process each window as it arrives and learn when to respond, with recent systems adding disentangled perception–decision loops.The cited approaches include systems that refine streaming processing through separate perception and decision components.
  • Streaming Video Understanding: OVO-Bench measures real-time perception, backward tracing, and forward active responding, while StreamingBench adds contextual understanding; both are used for evaluation.These benchmarks characterize several capabilities in streaming video understanding.
  • Streaming Video Understanding: Streaming models maintain compact state for the immediate present, whereas GROVE retains structured memory that remains queryable over days.This is the central distinction drawn between prior streaming systems and GROVE.

Methodology

GROVE builds a causally updated, temporally stratified memory from streaming video and exposes scale-native retrieval skills for both reactive question answering and proactive assistance. Its memory grows through dual perception, adaptive segmentation, moment extraction, and pattern consolidation without future-video access.

  • Memory construction: Each arriving window undergoes dual perception, producing dense narrative captions and a perceptual trace entry before adaptive segmentation groups captions into activities.Segmentation conservatively keeps ambiguous activities open until semantic change or a maximum-length cap triggers closure.
  • Memory construction: Closed episodes are atomized into timestamped moments and consolidated with temporally non-contiguous episodes into patterns that record recurrence, frequency, and typical time.Patterns capture regularities absent from any single episode, while moment-to-episode edges support recovery of surrounding context.
  • Causal updates: All memory updates use only observations available by time t, making the current window immediately available while moments, episodes, and patterns update causally after episode closure.No retrospective pass over future video is required.
  • Retrieval interface: Four scale-native skills—Perception Lookup, Moment Recall, Episode Replay, and Pattern Traversal—retrieve evidence at matching granularities rather than through one flat index.Episode Replay supports coarse-to-fine retrieval, while Pattern Traversal follows pattern-to-episode-to-moment group edges for broad or cross-period queries.
  • Agentic access: Reactive reasoning iteratively selects retrieval skills or answers from accumulated evidence, whereas proactive assistance uses the same memory interface to decide whether a service is needed and what to provide.Both modes obey the causal cutoff; proactive service families operate at moment, episode, or pattern scales.

Experiments

Across five video-memory benchmarks, GROVE achieves leading results for proactive assistance, lifelong QA, and online video understanding. Ablations, efficiency measurements, and qualitative cases show complementary temporal strata, scale-native retrieval, and live-stream-compatible construction.

  • EgoServe: GROVE achieves the best overall Macro-F1 on EgoServe at 12.6, exceeding EgoMemo by 4.6 points and ranking first on seven of ten service subtypes.Its largest gains are in safety (19.7 vs. 12.5), tool use (20.2 vs. 7.5), and error recovery (23.8 vs. 1.7).
  • OVO-Bench and StreamingBench: GROVE obtains the best OVO-Bench overall score of 67.5 and StreamingBench average of 65.1 among the compared models.It leads OVO real-time perception at 76.2 and scores 53.1 on StreamingBench contextual understanding, 9.2 points above the strongest listed baseline.
  • Ablations: Removing the EPISODE layer causes the largest day-scale ablation drop, from 18.75 to 14.57, while temporal-range constraints improve localization on both day and week splits.Removing the constraint reduces scores from 18.75 to 15.08 on day and from 19.75 to 13.50 on week.
  • Ablations: Grounding proactive retrieval in the current perceptual trace outperforms an LLM-written query on EgoServe, scoring 12.62 versus 12.17.The perceptual trace grounds retrieval in observed entities and actions rather than free-form query generation.
  • Efficiency: Build time per minute of video drops from 78.2 to 19.5 seconds, exceeding real time from one frame every five seconds onward and reaching a 3.08× real-time factor.Captioning, episode segmentation, and moment/pattern consolidation run as three overlapping asynchronous stages.
  • Qualitative analysis: Qualitative cases demonstrate retrieval at the appropriate scale for fine-grained detail, procedural replay, and query-free pattern traversal.The examples locate a hotpot episode, recover a cooking episode’s next step, and connect Day-4 assembly with the Day-1 session.

Conclusion

GROVE is a training-free framework that organizes streaming-video memory and retrieval around temporal scale. It supports reactive question answering and proactive assistance through a shared memory and specialized skill library.

  • Memory organization: GROVE causally transforms incoming observations into perceptual traces, time-stamped moments, coherent episodes, and recurring patterns.Each temporal stratum has a retrieval skill specialized for its scale.
  • Unified access: Reactive question answering and proactive assistance share GROVE’s memory and skill library.The framework unifies both behaviors through the same access infrastructure.
  • Evaluation: Results across five benchmarks and controlled ablations support pairing temporal structure with scale-specific access.The supplied conclusion passage states that both benchmark results and ablations show this pairing is effective.

Limitations

GROVE has limitations from frozen perception, delayed higher-level updates, growing computational costs, and benchmark differences in reasoning backbones. Missed evidence cannot be recovered, and current-window queries depend on the perceptual trace until episodes close.

  • Perception and ingestion: GROVE inherits errors from its frozen perception model, and evidence missed during ingestion cannot be recovered through later consolidation.Perceptual omissions remain unavailable to subsequent memory consolidation.
  • Memory updates: Higher-level memory updates only when an episode closes, so current-window queries rely on the perceptual trace.This creates a temporal gap before newly observed content reaches higher-level memory.
  • Scalability: Memory construction and multi-round retrieval become more costly as the observed history grows.The limitation affects both building memory and repeatedly accessing it.
  • Evaluation: Because the main benchmarks use different reasoning backbones, component-level conclusions rely on fixed-backbone ablations.This makes backbone variation a constraint on interpreting component-level findings.

A. Methods · A.1 Details on Memory Construction · A.2 Retrieval Details on Different Benchmarks

GROVE constructs a causally updated, queryable memory by retaining perceptual traces and consolidating them into moments, episodes, and patterns. Retrieval is benchmark-specific: some tasks use agent-selected skill calls under budgets, while others use fixed schemas or pre-injected pattern context.

  • A.1 Details on Memory Construction: Pattern matching examines candidates in parallel batches of 10 and stops after the first affirmative match.When a pattern is re-synthesized, only its five most recent episodes contribute full summaries; older episodes are compressed into short subjects.
  • A.1 Details on Memory Construction: The memory graph links patterns to grouped episodes, episodes to weighted moments, and moments to perceptual traces, preserving literal counts and on-screen text that summaries can blur.Patterns can span multiple days, while the strata differ in kind rather than only in length.
  • A.1 Details on Memory Construction: 9.4–15.2 moments per episode show that the moment stratum is consistently an order of magnitude larger because it retains second-level evidence.The ratio ranges from 9.4 on StreamingBench to 15.2 on MM-Lifelong.
  • A.1 Details on Memory Construction: 6,786 episodes consolidate into 2,041 patterns on MM-Lifelong, while 8,002 episodes consolidate into 2,531 patterns on EgoServe.Both whole-recording settings yield about 3.2 episodes per pattern, compared with barely two episodes per pattern on clip-level benchmarks.
  • A.1 Details on Memory Construction: The pattern layer grows sub-linearly in long-horizon recordings, keeping the entry point for broad questions small after weeks of recording.Short clips rarely revisit the same activity, so their patterns group barely two episodes each.
  • A.1 Details on Memory Construction: Each arriving video window is processed by a VLM into a perceptual trace that is immediately queryable.The online state maintains R, F, E, and P, producing a memory queryable at any time t.
  • A.1 Details on Memory Construction: Recording gaps over 60 seconds or day changes deterministically cut episodes, while insufficient frames extend the current unit until a length guard forces a cut.Closed episodes trigger parallel feature extraction, pattern matching, updating, or creation before the next unit opens.
  • A.2 Retrieval Details on Different Benchmarks: MM-Lifelong, OVO-Bench, and StreamingBench let agents select retrieval calls under multi-round budgets, whereas ESTP-Bench and EgoServe use fixed-schema retrieval without a query or forward request.On MM-Lifelong, day/week/month splits use round budgets of 8/6/12 and inject pattern maps capped at 46/100 entries for day/week, or the full timeline for month.

Skill-based Agentic Reasoning · Schema-based Proactive Assistance · B. More Experimental Results

GROVE’s agentic reasoning selectively invokes scale-appropriate retrieval skills based on question needs and answer-model routing, while proactive assistance uses schema-driven evidence assembly and dataset-specific screening configurations. Across settings, the system combines recent captions, episode summaries, and temporal memory strata with different decision schedules and service-merging rules.

  • Skill-based Agentic Reasoning: On MM-Lifelong, replacing gpt-5-mini with gpt-5.2 expands day and week from four to eight invoked skills, with R calls rising from 0% to 51% and 26%.The memory, prompt, and call library remain fixed; search_ocr becomes the most frequent skill.
  • Skill-based Agentic Reasoning: OVO-Bench backward questions use the EPISODE skeleton and temporal strata, whereas forward questions rely on recent frame captions, retrieved MOMENTs, and previous answers while deciding whether to respond or wait.The same solver and one-tool-document setup support the real-time and backward tracks, while forward evaluation emphasizes response timing.
  • Skill-based Agentic Reasoning: The agent’s MM-Lifelong skill profile is question-dependent: search_entities averages 0.28 calls, search_episode_moment 0.23, search_moment_only 0.11, and get_recent_episodes 0.07 per question.The reported call frequencies describe the skills actually invoked rather than every exposed capability.
  • Skill-based Agentic Reasoning: On OVO-Bench real-time, the agent averages 0.81 get_recent_caption calls per question, invokes semantic search zero times across 2,500 questions, and answers roughly one in five questions without a call.The real-time track uses a chronological EPISODE-summary skeleton as starting context, with a three-round, one-call-per-round budget.
  • Skill-based Agentic Reasoning: ESTP-Bench performs retrieval through the schema rather than agent-issued calls, combining current-step captions, interaction history, and MOMENT plus perceptual-trace evidence when past grounding is needed.The system decides when and how to answer a standing question during the stream.
  • Schema-based Proactive Assistance: Schema-based proactive assistance revisits decisions every second-level caption for single queries, while conversational turns use short windows beginning when each turn is asked and denser revisits for task-progress answers.Both settings carry prior interaction context into their decision process.

B.1 Extended Results of the Main Paper

GROVE achieves the best score on every MM-Lifelong split, including the month-scale training videos. Ablations show that temporal memory strata and scale-native retrieval skills contribute complementary capabilities, especially for long-range assistance.

  • Full Comparison on MM-Lifelong: GROVE attains the best score on every MM-Lifelong split, retaining its margin over ReMA on month-scale videos at 18.98 vs. 17.62.The result suggests the gains are tied to memory organization and retrieval rather than a single evaluation split.
  • Retrieval Skills: Episode Replay is essential for coarse-grained activity reasoning, reducing memory recall from 5.9 →1.1 and routine optimization from 7.1 →4.0 when removed.A recalled fact becomes actionable only when the surrounding activity can be read.
  • Retrieval Skills: Without Pattern Traversal, memory recall falls to 2.7 and routine optimization to 4.2, exactly matching deletion of the PATTERN stratum on this benchmark.The two interventions therefore coincide for these outcomes.

B.2 Additional Experimental Results

Additional experiments show that GROVE’s performance depends on retrieval width, adaptive episode boundaries, and flexible skill selection. Narrow retrieval, fixed segmentation, and single-stratum retrieval choices can reduce accuracy or misalign evidence with the activity being recognized.

  • Retrieval Width: At k = 3, every split loses three to five points because relevant evidence is often absent from the first few ranked items.Accuracy peaks at k = 20 on the day and week splits before degrading on day (18.75 →16.33) as redundant items dilute the evidence.
  • Skill Selection: Random alternation across retrieval skills reaches 16.00, outperforming the best single-stratum baseline at 13.00.Random-3 reaches 20.75 on day but only 15.97 on month, while its day advantage comes from inflating the prompt evidence block by ∼40% in prompt tokens.
  • Episode Segmentation: Removing adaptive episode segmentation reduces performance from 12.62 to 12.17 while keeping screening windows, segment counts, and evidence volume unchanged.The ablation changes only where episode boundaries fall, isolating boundary placement as the source of the performance loss.
  • Episode Segmentation: Fixed-length units mix two activities and shift the trigger second away from the event, even when prediction volume remains unchanged.The resulting summaries no longer land inside the tolerance window because their boundaries do not align with the activity.

C. Case Study

GROVE answers a temporal-ordering question by progressively searching episodes, moments, and fine-grained time ranges across a 24.5-hour recording. Reconstructing timestamps rather than recalling directly yields the correct boss-defeat order 1, 4, 2, 3.

  • Case Study: The case involves four bosses defeated across a 32-minute span within a 24.5-hour recording, preventing any single call from covering them all.This temporal separation requires multi-stage retrieval.
  • Case Study: GROVE first locates the relevant region with search_episodes, then retrieves individual defeat records using search_moments and P→E→F traversal.The process unfolds over rounds 1–3 before narrowing to exposed time ranges.
  • Case Study: The system narrows search_moments to [01:20:00, 01:25:00] until every boss has a timestamp, reconstructing the timeline and producing order 1, 4, 2, 3.The answer is read from the reconstructed timeline rather than recalled.
Loading 2608.02392v2…