PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoProAgent: Toward Memory-Driven Proactive Assistance

Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen, Tammy Stark, Xianhui Zhu, Yuetai Li, Rulin Shao, Tianxin Wei, Rakesh Wanga, Anuj Kumar, Jiliang Tang, Wen-tau Yih, Xin Luna Dong

arXiv:2610.22671v2cs.CL

TL;DR

Memory-driven proactive assistance requires finding the personal-memory evidence that warrants an intervention and deciding when to speak, rather than answering an explicit query. MemoProBench measures this jointly, and MemoProAgent improves strict accuracy, with further gains from lightweight RL post-training.

  • Problem

    Prior proactive-assistance settings provide the fact that justifies an intervention, leaving open how systems perform when they must find it in long-term personal memory without a query.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MemoProBench, pairing egocentric video with year-long personal memories and human-verified trigger windows, and builds MemoProAgent with graph-organized memory retrieval and an interruption gate.

    Source: §1 · pp. 2–3

  • Results

    MemoProAgent raises strict accuracy from below 10% to 37.5% on an open-source model and 38% on GPT-5.3-Chat without training; lightweight RL post-training reaches 42.4%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Controlled tests show that correct memory evidence and correct timing are independently necessary for proactive assistance.

    Source: §1 · p. 3

    MemoProBench uses synthesized text-based histories rather than recorded multimodal histories; extending it to naturalistic multimodal histories is future work.

    Source: Limitations · p. 14

Abstract

from arXiv · show

As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce MemoProBench, a benchmark for memory-driven proactive assistance (MPA), pairing egocentric video with year-long, distractor-rich user memories (~618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every baseline we evaluate performs poorly (at most 15%), including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window GPT-5.3-Chat reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build MemoProAgent with a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. Without any training, MemoProAgent lifts strict accuracy from below 10% to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat; RL post-training reaches 42.4%.

1 Introduction

Memory-driven proactive assistance requires a system to find the memory that warrants an intervention and decide when to speak, without an explicit query. MemoProBench measures this challenge, and MemoProAgent addresses it with memory retrieval and gating, improving strict accuracy.

1 Introduction

  • Prior proactive-assistance settings provide the intervention-justifying fact, whereas this setting requires finding it in long-term memory without a query.Memory also determines whether to intervene at all, not only what to say.Source: §1 · pp. 1–2
  • A controlled study finds that oracle evidence delivered in-window yields 96.0% for GPT-5.3-Chat, versus 38.0% when it chooses timing and 15.3% with retrieved evidence.The study identifies memory content and timing as independently necessary.Source: §1 · p. 3
  • MemoProBench pairs egocentric video with distractor-rich, year-long personal memories and scores whether, when, and what an assistant says.Samples have human-verified trigger windows and grounded reminders; clips where silence is warranted measure abstention.Source: §1 · pp. 2–3
  • 37.5% strict accuracy without training on an open-source model and 38% on GPT-5.3-Chat; lightweight RL post-training reaches 42.4%.MemoProAgent uses a graph-organized memory harness that surfaces relevant memories and gates out evidence that does not warrant interruption.Source: §1 · p. 3

2 Related Work

Prior memory-augmentation systems retrieve context to answer explicit queries, while most proactive-assistance work supplies the evidence for an intervention. MemoProProBench instead requires retrieval from personal memory and joint evaluation of content and timing.

2 Related Work

  • Memory augmentation retrieves context for an explicit request, but memory-driven proactive assistance must find evidence as the user's surroundings unfold, before deciding whether to intervene.The paper distinguishes its query-free setting from systems that use memory to answer or ground a request.Source: §2 · p. 3
  • Most prior proactive settings supply the evidence licensing a response and primarily evaluate timing, whereas this work requires retrieval and jointly grades content and timing.The supplied evidence can take the form of an explicit question, goal, or short profile; the paper contrasts these settings with long personal memory.Source: §2 · p. 3
  • Compared with concurrent EgoServe, MemoProBench uses controlled memory to isolate why systems fail, while MemoProAgent adds a gate and post-training to mitigate distracting-memory failures.The passage identifies Gong et al. (2026) as concurrent work and describes its EgoMemo agent as training-free.Source: §2 · p. 4

3 The MemoProBench Benchmark

MemoProBench evaluates whether an assistant can use long-term personal memory to deliver a grounded intervention at the right time, or remain silent when none is warranted. It formalizes the task, describes the benchmark data, and defines joint content-and-timing metrics.

3.1 Problem Definition

  • The task is an always-on, causal, query-free policy over recent video and long-term memory, emitting a natural-language intervention or silence.Memory organization and access are left to the evaluated method.Source: §3.1 · p. 4
  • A correct intervention must be grounded in the relevant memory and current frame, occur in the trigger window, and address its earliest appropriate appearance exactly once.Repeated firing, or firing outside the window, is incorrect; later reappearances do not create a new opportunity for the same matter.Source: §3.1 · p. 5

3.2 Data Statistics

  • MemoProBench pairs egocentric clips with year-long personal histories averaging ~618K tokens; the target evidence is ~0.19% of each history, and 150 negative clips require silence.The 1,804 pairs include 904 training, 100 development, and 800 evaluation examples; each clip has one trigger window.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • Perfect requires one fire with good content in the allowed interval; Hit counts any such fire, while early-, late-, and over-trigger measures capture distinct errors.The evaluation also reports average fires per clip; its default timing tolerance is 2 seconds.Source: §3.3 · p. 6

4 Preliminary Findings

The preliminary study tests whether memory alone supports proactive assistance and separates the need for correct evidence from the need to choose when to speak. Across the two online visual-context paradigms, straightforward memory provision performs poorly, while the controlled study finds both evidence quality and timing necessary.

4 Preliminary Findings

  • The preliminary study compares ways of supplying memory, then varies evidence quality and timing independently across a 2 × 2 controlled study.Timing is either supplied by exposing evidence only inside the trigger window or left to the model, which receives evidence at every decision step.Source: §4 · pp. 6–7, §4.3 · p. 8

4.1 Proactive Paradigms

  • The sliding-window paradigm processes the four most recent frames independently every two seconds, while streaming appends one frame per second to persistent visual context.The streaming context retains at most 30 turns before starting a new chunk; it provides a longer view but a longer, noisier context.Source: §4.1 · pp. 7–8

4.2 RQ1: Does having the memory suffice?

  • Neither full history, a summary, nor retrieved fragments make the task usable: even the strongest frontier baseline reaches only 15.0% Perfect.For Qwen3.8-27B, Standard RAG reaches 42.5% Hit but only 7.0% Perfect, while full-history GPT-5.3-Chat has 83.9% Hit and 99.9% over-triggering.Source: §4.2 · p. 8

4.3 RQ2: If not, what is missing — the content, the timing, or both?

  • With oracle evidence and timing given, Perfect is 77.3% for Qwen3.8-27B and 96.0% for GPT-5.3-Chat; leaving timing to the models lowers these to 26.0% and 38.0%.With Standard RAG and timing given, scores are 7.3% and 15.3%, respectively; without timing, they are 5.3% for Qwen3.8-27B and 8.0% for GPT-5.3-Chat.Source: §4.3 · p. 9, Table 4 · p. 9

5 Method

MemoProAgent pairs structured memory retrieval with a gate that surfaces evidence only when it warrants intervention. A lightweight RL stage adapts the generator to this harness and rewards correct timing, content, and silence.

5.1 The MemoProAgent Harness

  • MemoProAgent organizes episodic and semantic memories in a graph, retrieves facts related to observed objects, and gates them into one refined evidence item—or abstains—when they justify intervention.The harness leaves the existing model weights unchanged and supports both streaming and windowed paradigms.Source: §5.1 · pp. 9–10
  • The generator uses a two-round observe–retrieve–reason–answer loop, deciding from the current view and returned evidence whether to give a grounded reminder or stay silent.The gate filters memories that may be real but are not a good reason to intervene, such as resolved open loops.Source: §5.1 · p. 10

5.2 Post-Training

  • RL post-training gives positive reward for correct silence or a correctly timed, high-quality reminder, and penalizes misses, false fires, low-quality reminders, and malformed outputs.GRPO training uses four trajectories per state; the reward also penalizes repeated in-window interventions as mistimed fires.Source: §5.2 · pp. 10–11

6 Experimental Results

MemoProAgent improves strict accuracy over memory-augmentation baselines, and RL further improves performance, particularly by reducing premature and excessive firing. Further probing reports transfer to Epic-Kitchens, fewer fires on negative clips, and contributions from the gate, graph memory, and intervention history.

6.1 Experimental Setup

  • The experiments compare windowed and streaming paradigms, using Qwen3.8-27B and GPT-5.3-Chat for the harness and multiple backbones and baselines for comparison.The primary metric is Perfect; Hit, over-, early-, and late-trigger rates and fires per clip provide breakdowns.Source: §6.1 · pp. 11–12

6.2 Main Results

  • As Standard RAG retrieves more events, Hit rises modestly while over-triggering and fires per clip rise sharply, causing Perfect to collapse.MemoProAgent maintains relatively high Hit with low over-triggering.Source: Figure 6 · p. 12, §6.2 · p. 13
  • 42.4% Perfect for windowed Qwen3.8-27B after RL post-training, up from 37.5% without training and 7.0% with best-k Standard RAG; streaming reaches 40.6%, up from 33.3%.RL lowers over-triggering from 22.3% to 13.5% windowed and from 32.8% to 25.3% streaming; GPT-5.3-Chat reaches 38% Perfect with the inference-only harness.Source: §6.2 · p. 12

6.3 Further Probing

  • On 100 unseen Epic-Kitchens samples, the harness reaches 41.0% windowed and 37.0% streaming Perfect, while the Ego4D-trained generator reaches 49.0% and 41.0%.On 150 negative clips, MemoProAgent fires on 15.3%, compared with 77.3%/84.0% for Standard RAG; RL lowers this to 14.0%/14.7%.Source: Figure 7 · p. 12, §6.3 · p. 13
  • Removing the gate costs 13.3 Perfect points, replacing graph memory with Standard RAG costs 16.6 points, and hiding prior responses drops Perfect to 8.0.The same ablation trends hold in streaming.Source: §6.3 · p. 13

7 Conclusions

The paper frames memory-driven proactive assistance as requiring the right evidence at the right time, then presents MemoProAgent as a response to that joint challenge. Its harness improves strict accuracy without changing model weights, and lightweight RL post-training raises it further.

7 Conclusions

  • Memory-driven proactive assistance asks memory to determine not only what an assistant says, but whether and when it should speak.The paper’s conclusion identifies the need to deliver the right memory at the right time as the task’s central demand.Source: §7 · p. 13
  • The controlled study finds that correct memory and correct timing are independently necessary, while MemoProAgent raises strict accuracy from below 10% to 37.5% without weight changes and to 42.4% with RL.The controlled study reports a 96% oracle ceiling and performance under 40% when timing or retrieved evidence replaces an oracle condition.Source: §7 · p. 13

Limitations

MemoProBench uses synthesized, text-based caption histories and single-trigger clips; it does not evaluate latency.

Limitations

  • The benchmark uses synthesized caption streams rather than recorded multimodal histories, leaving naturalistic multimodal histories for future work.Synthesis enables controlled trigger windows and exact justifying evidence.Source: Limitations · p. 14
  • The evaluation measures task performance, not latency; system-level optimizations are left to future work.Production latency varies with hardware, infrastructure, and network conditions.Source: Limitations · p. 14
  • Each clip contains one trigger window; richer scenarios with multiple, diverse triggers and adversarial memory remain future extensions.Source: Limitations · p. 14

AI Use Statement

The authors used generative AI to construct synthetic benchmark data and polish the paper, while retaining responsibility for the final content.

AI Use Statement

  • Generative AI tools proposed memory evidence, synthesized personas and year-long timelines, and rendered events into caption streams.All included samples passed model-based quality control and independent review by 32 trained annotators; only human-accepted samples were retained.Source: AI Use Statement · p. 14
  • The authors also used generative AI to improve grammar, clarity, and readability, then reviewed the edits for technical accuracy.Source: AI Use Statement · p. 14
  • The authors take responsibility for the paper’s final content, including AI-assisted text, claims, and artifacts.Source: AI Use Statement · p. 14

Ethics Statement

MemoProAgent uses a two-round decision process to retrieve and gate relevant memory before deciding whether to issue a reminder. The appendices describe its windowed and streaming implementations and baseline input strategies.

Ethics Statement

  • MemoProBench studies when an assistant should speak and the interruption cost of speaking wrongly, while noting deployment privacy considerations for continuous observation and memory.Its histories are synthesized and do not correspond to real individuals; 32 trained annotators verified trigger windows and rationales.Source: Ethics Statement · p. 14

A.1 Overview

  • MemoProAgent identifies objects in focus, retrieves object-matched episodic and semantic memories, gates them to one actionable fact or abstains, then emits a grounded reminder or NO_REMINDER.Source: §A.1 · p. 17

A.2 Windowed Implementation Details

  • In windowed mode, decisions occur every 2 seconds using the current frame and three preceding frames sampled at 1-second intervals.The second round receives gated evidence and previously spoken reminders; each visual window is processed independently.Source: §A.2 · pp. 17–18

A.3 Streaming Implementation Details

  • Streaming mode makes decisions every second, appending each latest frame to retained visual context and applying the same two-round order.At the 30-turn limit, it keeps the latest four turns and starts a new context chunk.Source: §A.3 · p. 18

B Baseline Implementation Details

  • The baselines differ in memory exposure: None omits long-term memory, All-history supplies chronological history, Summary supplies a compact summary, and Standard RAG retrieves raw passages without MemoProAgent’s gate.Source: B Baseline Implementation Details · p. 19

B.1 Windowed Baselines

  • Windowed None, All-history, and Summary baselines directly decide from the current and preceding frames; Standard RAG instead retrieves top-k raw passages without gating before the decision.Previously fired reminders are supplied through Already spoken.Source: §B.1 · p. 19

B.2 Streaming Baselines

  • Streaming baselines retain visual-dialogue context; Qwen Standard RAG retrieves ungated raw passages, while VideoLLM-online and MMDuet2 retain their native streaming prompts and state.For Qwen, only the final short answer is retained for later decisions.Source: §B.2 · pp. 20–21

C Data Construction Pipeline

MemoProBench constructs each instance by aligning a video trigger, justifying memory evidence, and a plausible long-term personal history. Model checks and human review produce verified benchmark instances.

C Data Construction Pipeline

  • The pipeline scans the target object across the clip, merges visible segments separated by at most 8 seconds, and uses the first and last timestamps to define a trigger window.The pipeline figure depicts this localization followed by evidence generation, timeline construction, and quality checks.Source: C Data Construction Pipeline · p. 22, Figure 8 · p. 22
  • For each target, an LLM proposes episodic or semantic memory evidence, which a worth-triggering gate checks for grounding, target specificity, and reminder warrant.Source: C Data Construction Pipeline · p. 22
  • The evidence is embedded in a persona-conditioned year-long timeline of milestones, interests, daily distractors, and evidence, rendered as timestamped first-person action and conversation captions.This makes the relevant memory a small portion of a coherent long-horizon history.Source: C Data Construction Pipeline · p. 22
  • Model quality control checks grounding, target specificity, warrant, and unrelated-object contamination before human review accepts, fixes, or rejects samples.Source: C Data Construction Pipeline · p. 22
  • The final benchmark contains 1,804 human-verified instances, with timestamped long-term memories averaging 618K tokens.Source: C Data Construction Pipeline · p. 23

C.1 Human Annotation Details

  • Annotators assessed visibility, trigger-window correctness, reminder warrant, and memory-event consistency; 82.7% of windows were kept, 14.2% adjusted, and 3.1% unusable.Majority-vote review accepted 1,804 samples, 83.8% of reviewed samples.Source: §C.1 · p. 23

D.1 Data Statistics

The evaluation set spans varied everyday scenes and reminder reasons, with trigger windows much shorter than the surrounding videos. The accompanying examples and analyses illustrate how long caption streams preserve target evidence that broad summaries can omit; streaming results reproduce earlier conclusions, and MemoProAgent improves frontier-model performance.

D.1 Data Statistics

  • Trigger windows are consistently much shorter than full videos across scenes and reminder reasons.The distribution varies across both scene types and reminder reasons.Source: §D.1 · p. 23

D.2 Examples

  • The examples pair justifying user memories with reminders and nearby video frames, illustrating varied scenes, memory types, and proactive intents.The evaluation examples include episodic memories and semantic memories.Source: §D.2 · p. 23, Figure 10 · p. 25

D.3 Caption Stream History Examples

  • The caption stream combines main-arc milestones, persona-derived interests, neutral daily distractors, and the single memory that justifies a reminder.Events are timestamped first-person captions, with [V] marking visible actions and [C] conversational content.Source: §D.3 · p. 27

D.4 Summary vs. Memory: Open-Loop Retrieval

  • The complete user summary captures broad persona traits but omits the consumed apples, empty supply, and intention to restock.The example contrasts a broad generated summary with the relevant memory event.Source: §D.4 · p. 28

E Preliminary Study under the Streaming Paradigm

  • The streaming-paradigm replications reproduce the sliding-window conclusions from the earlier RQ1 and content-by-timing studies.Table 15 repeats RQ1, and Table 16 repeats the RQ2 content-by-timing study.Source: E Preliminary Study under the… · p. 29

F MemoProAgent on a Frontier Model

  • 38% Perfect rate with MemoProAgent versus at most 10% for straightforward memory baselines on GPT-5.3-Chat, mirroring gains on Qwen3.8-27B.The authors report that a strong frontier model still cannot solve MPA on its own without the right memory delivered at the right time.Source: F MemoProAgent on a Frontier Model · p. 29

G Post-Training Details

The appendix details the RL training setup, reminder-content judging, and the memory graph's construction and retrieval. It also points to full baseline breakdowns for both deployment paradigms.

G Post-Training Details

  • Training uses 3,465 timestamp-level states from 904 source clips, disjoint from the fixed evaluation UIDs, and applies one pass of GRPO optimization.The states include in-window fires, early fires, already-spoken states, and no-evidence guard states.Source: G Post-Training Details · p. 29

H LLM Judge Prompt

  • Only reminders judged GOOD by GPT-4o count as content-correct; the judge evaluates grounding, correctness, and usefulness.The judge receives the reminder, current frame or frames, and relevant memory.Source: H LLM Judge Prompt · p. 30

I.1 Memory Construction Pipeline

  • Episodic graph triples retain event passages as provenance, while semantic triples store reusable standing information; open-loop flags cue retrieval rather than automatically triggering intervention.Mirroring can make an intention reachable from a visible object, such as representing an intention to buy milk as the milk carton needing restocking.Source: §I.1 · p. 31

I.2 Retrieval at Inference Time

  • Object-matched entity nodes seed one-hop retrieval and personalized PageRank, which can surface related event passages beyond exact object-name matches.Ranking uses entity similarity, open-loop cues, and recency; graph paths also use co-occurrence, co-mention, and synonym edges.Source: §I.2 · p. 31

I.3 Graph Examples

  • Table 19 illustrates episodic and semantic memory graph construction and how information can later be reached from a visible object.Source: §I.3 · p. 31, Table 19 · p. 31

J Full Main-Result Tables

Related papers

Loading 2610.22671v2…