PaperlayerView exact revision on arXiv

Source-linked AI summary

An Agentic Framework for Proactive Robots with Procedural Assistance Memory

Dohyun Lee, Minho Park, Jeonghoon Park, Dongyoon Hwang, Junha Hyung, Sungwon Hwang, Byungkun Lee, Jaegul Choo

arXiv:2610.21211v1cs.RO

TL;DR

Proactive assistance is difficult because the same observed behavior can imply different needs across users, while many robots wait for explicit requests. This paper uses a structured, evolving memory to guide a VLM’s assistance decisions and a VLA’s execution. Across five evaluation days, the system improves proactive prediction and response timing, and its memory adapts to routine changes.

  • Problem

    The same observable behavior can imply different desired outcomes across users, and proactive intent remains underspecified in settings with limited robotics data.

    Source: §1 · p. 1

  • Method

    A VLM uses a finite-state Procedural Assistance Memory of activity phases, assistance rules, and transitions to select tasks for a VLA, while daily reflection updates the memory.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    Across five evaluation days, PAM improves proactive prediction and response timing, while structured memory adapts to changes in the user’s routine.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support structured, evolving memory as a practical approach to longitudinal robot personalization.

    Source: §5 · p. 9

    The evaluation is restricted to seen scenarios rather than novel objects, layouts, or users, and clarification requests are outside the framework.

    Source: §5 · p. 9

Abstract

from arXiv · show

Proactive robots should anticipate user needs and provide assistance before being explicitly asked. A straightforward approach is to infer human intentions from commonsense knowledge, but the same observed behavior can imply different needs across users, potentially leading to unwanted assistance. We observe that everyday routines in the real world exhibit recurring patterns, so learning these patterns can reduce ambiguity in proactive decisions. Based on this, we propose a memory-based agentic framework in which a VLM task predictor reads a Procedural Assistance Memory (PAM) alongside the current observation to decide when to assist and what task to perform, and a VLA executor performs the predicted task. PAM is structured as a finite-state machine over activity phases, per-state assistance rules, and event-driven transitions; a reflection VLM revises PAM from each day's interaction trace without weight updates or retraining. In our 5-day desk-assistance evaluation, memory-free baselines remain conservative, with limited recall on proactive tasks. Our system improves consistently with accumulated experience, achieving the highest precision, recall, and F1 and reducing first-fire delay from approximately 5 seconds on Day 1 to 1 second by Day 5. Our approach also adapts to changes in the user's routine through continued memory reflection, recovering from memory--routine misalignment after a single memory update. The project page is available at https://davian-robotics.github.io/PAM/

1 Introduction

Proactive robots must recognize when assistance would help, but the same observed behavior can reflect different needs across users. This work proposes a memory-based framework that uses a user’s activity routine to choose when and what to assist, then revises that memory from daily interactions.

1 Introduction

  • Prior proactive systems often focus on fixed-protocol tasks, while proactive intent is underspecified and end-to-end models face scarce robotics data across diverse scenarios.Source: §1 · pp. 1–2
  • A fixed pretrained predictor cannot resolve cases where the same observable behavior implies different desired outcomes across users because the missing information is in individual habits.Figure 1 illustrates visually similar observations mapping to different correct actions.Source: §1 · p. 2, Figure 1 · p. 2
  • PAM records a user’s routine as activity-phase states, transitions, and per-state assistance entries for deciding when and what to assist.The memory is updated during deployment.Source: §1 · p. 2
  • The framework conditions a high-level VLM on user-specific routine context to decide when to assist and what task to issue without explicit requests.Source: §1 · p. 2
  • A self-evolving memory loop revises PAM offline after each day using observed behavior and user feedback, without retraining.The authors describe this as enabling learning of routines, adaptation to changes, and reuse of feedback.Source: §1 · p. 2

2 Related Work

Prior work separates high-level reasoning from low-level action execution and studies proactive assistance through structured task progress, daily object-use patterns, or momentary intent cues. This paper situates user-specific evolving memory as a way to retain the activity context needed for proactive decisions.

2.1 Vision-Language-Action Models

  • System 1–2 designs separate a reasoning module that interprets scenes and plans from an executor that generates low-level robot-control actions.The paper uses a VLM for proactive task prediction and a VLA for executing selected atomic skills.Source: §2.1 · p. 3

2.2 Proactive Assistance in Embodied HRI

  • Structured-task methods estimate progress when subtasks and assistance opportunities are known, while object-usage methods typically require substantial logs or curated observations.These requirements can limit flexibility across diverse routines and changing user behavior.Source: §2.2 · p. 3
  • Intent-inference systems use verbal or non-verbal cues, but a momentary cue can imply different needs depending on the user, context, and activity history.The paper conditions proactive decisions on user-specific memory accumulated across days.Source: §2.2 · p. 3

2.3 Evolving Memory for Test-Time Adaptation

  • Prior external-memory agents support test-time adaptation, but their memories generally target text-centric agents, games, or instruction-following rather than proactive robot assistance.PAM organizes assistive opportunities by activity phase, per-state entries, and transitions.Source: §2.3 · p. 3

3 Method

The method pairs real-time task prediction and execution with a structured memory of the user’s routine. A daily reflection process revises that memory from recorded behavior and feedback, while it remains fixed during each day.

3 Method

  • The framework uses observations and PAM to predict whether assistance is needed, selects a task for a VLA executor, and updates PAM offline from recorded video after each day.Figure 2 depicts the task-prediction and memory-reflection paths.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Real-Time Task Prediction and Execution via System 1–2 Architecture

  • At each 1-Hz step, the task predictor reads recent video frames, PAM, and a fixed instruction, then outputs a natural-language task or NO_ACT.The VLA executor generates low-level actions for the selected task.Source: §3.1 · p. 4

3.2 Procedural Assistance Memory (PAM)

  • PAM uses activity-phase states, per-state trigger–action pairs, and transitions to contextualize cues; a trigger should precede an action the robot can perform for the user.For example, sitting with an empty bowl can trigger bringing cereal before the user fetches it.Source: §3.2 · p. 4, Figure 3 · p. 4
  • The predictor reads PAM’s state descriptions, trigger–action pairs, and transitions alongside recent observations, choosing only immediately executable assistance and otherwise outputting NO_ACT.Source: §3.2 · p. 4
  • The evaluation organizes six proactive tasks as a continuous desk-activity sequence, from delivering cereal or milk to storing items and discarding a used wipe.Source: Figure 4 · p. 5

3.3 Offline Memory Reflection and Update

  • At day’s end, the reflector reviews sampled video, prior PAM, action history, and available feedback to identify missed opportunities and recurring patterns, then revises rules or transitions.PAM remains fixed within each day; the updated memory is used the following day.Source: §3.3 · p. 5

4 Experiments

The experiments test proactive prediction over five days, memory adaptation after a routine change, and task execution against monolithic VLAs. Across these evaluations, FSM memory improves proactive prediction over memory-free and list-memory alternatives, and the separated System 1–2 framework outperforms monolithic variants on task execution.

4.1 Experimental Setup

  • The five-day morning-desk study labels six assistance tasks and NO_ACT per frame, reporting macro precision, recall, F1, and first-fire delay.The routine includes preparing items for the user’s next activity and clearing them after use, so the same objects can require different actions as the routine progresses.Source: §4.1 · pp. 5–6

4.2 Proactive Prediction across Memory Designs

  • Ours w/FSM Memory achieves the highest precision, recall, and F1 on T1–T6, with first-fire delay falling from roughly 5 seconds on Day 1 to about 1 second by Day 5.Memory-free baselines remain conservative, while List Memory saturates early; FSM Memory’s eagerness lowers NO_ACT recall, but its higher NO_ACT precision keeps NO_ACT F1 on par with the baselines.Source: Figure 5 · p. 6, §4.2 · p. 7

4.3 Memory Adaptation and Evolution

  • After one reflection, A-Evolving’s recall and F1 reach levels comparable to Scratch-Evolving on Day 2, while reflection revises FSM rules and transitions as routines change.Across Days 3–5, missing trigger-action pairs are recovered and per-state rules are completed; in the changed routine, a cereal-delivery trigger is revised to vitamin delivery.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 Comparison with Monolithic VLAs

  • Monolithic π0.5 variants have lower atomic-task success than System 1–2; under low data, joint reasoning and control plus misleading visible objects compound execution errors.The comparison includes π0.5 with and without subtask reasoning; the framework delegates atomic action generation to π0.5 and reasoning to Qwen 3.5.Source: §4.4 · pp. 8–9, Table 1 · p. 9, Figure 8 · p. 9

5 Conclusion

PAM personalizes proactive assistance through daily reflection without weight updates, and the conclusion reports improved prediction and response timing across five evaluation days. The paper also identifies limits in ambiguous-trigger handling and evaluation scope.

5 Conclusion

  • Across five evaluation days, PAM improves proactive prediction and response timing, while ablations support FSM structure, continued reflection, and System 1–2 decomposition.The conclusion describes structured, evolving memory as a practical approach to longitudinal robot personalization.Source: §5 · p. 9
  • Clarification is outside the framework when a learned trigger is handled briefly without commitment; evaluation also excludes novel objects, layouts, and users.The authors attribute the latter boundary to limited capacity of current open-source VLAs.Source: §5 · p. 9

Appendix

  • The appendix covers setup and annotation, per-task proactiveness, memory-update analysis, another routine, reverse routine change, and predictor and reflector prompts.Figure 9 shows the same canonical desk routine across five evaluation days.Source: Appendix · p. 14, Figure 9 · p. 15

A.1 Evaluation Dataset and Annotation

The evaluation dataset comprises five days of one user performing a recurring desk routine, with per-frame labels for six proactive tasks and NO_ACT. Natural variation in timing and desk state lets the authors examine prediction over repeated exposure.

A.1 Evaluation Dataset and Annotation

  • Five days of video capture one user repeating the same desk routine, with action timing and visible desk state varying naturally across days.The routine includes cereal and milk, eating, laptop work, wiping the mouth, and leaving the desk.Source: §A.1 · p. 15
  • The authors label 1,146 sampled frames across five days, assigning each one of six proactive-task labels or NO_ACT.Recordings last roughly 3.5–4.5 minutes per day; frames are sampled uniformly at 1 fps, yielding 197–270 frames per day after deletions.Source: §A.1 · p. 15
  • A task label marks the most natural proactive intervention given the scene and ongoing behavior; all other frames receive NO_ACT.The annotations serve as ground truth for proactive-task prediction on this routine.Source: §A.1 · p. 15

A.2 Evaluation Settings

The evaluation uses a frame-based task predictor and an end-of-day memory reflector, with their visual inputs and call procedures specified. Annotation statistics define the six task categories and NO_ACT.

A.2 Evaluation Settings

  • The predictor receives five recent frames sampled at two-second strides, current memory, and the final frame’s wall-clock time; its calls average approximately 1 second.It sees the two back-facing camera views, outputs a JSON object, and uses the same Qwen 3.5–27B backbone as the reflector.Source: §A.2 · pp. 15–16
  • The task categories are six proactive actions—T1 through T6—and NO_ACT for frames where no intervention is appropriate.The table identifies T1–T6 as delivering cereal or milk, storing cereal or milk, storing the bowl, and discarding the used wet wipe.Source: Table 2 · p. 16
  • Once per day, the reflector processes 200 downsampled frames with prior memory, an action-history summary, and any user feedback, then emits updated memory.Thinking mode is enabled, and calls average approximately 1 minute.Source: §A.2 · p. 16

A.3 Training and Deployment for VLAs

The VLA executor is trained for individual instructed tasks, while monolithic baselines learn when and what to do from full-scenario demonstrations. The hardware setup gives VLAs wrist-camera grasp context in addition to the two back views.

A.3 Training and Deployment for VLAs

  • The executor is fine-tuned on 10 teleoperation episodes per task, whereas the monolithic baseline uses 10 full-scenario episodes with a fixed high-level instruction.The monolithic + reasoning baseline uses the same full-scenario episodes and adds per-frame subtask supervision, predicting a subtask before conditioning its action.Source: §A.3 · pp. 16–17
  • All three VLA variants share a 50,000-step π0.5 fine-tuning recipe, and the action head predicts 50-step chunks executed before requesting another chunk.Training uses AdamW and an effective batch size of 64; deployment uses 10 denoising steps per chunk.Source: §A.3 · p. 17

A.4 Hardware Setup

  • The FR3 robot uses three RGB cameras: two back-facing views feed the VLM stack, while the VLAs also receive a wrist view for close-up grasp context.The robot is a 7-DoF manipulator; the control loop runs at 20 Hz, and demonstrations use a VR teleoperation interface.Source: §A.4 · pp. 17–18, Figure 10 · p. 17

A.3 Training and Deployment for VLAs

  • Memory-based approaches outperform memory-free baselines on Milk→Person and Cereal→Storage while maintaining comparable NO_ACT F1; FSM Memory adds gains on Cereal→Storage and Milk→Storage.These per-task comparisons are reported in Figure 11.Source: Figure 11 · p. 18

B Per-task Results of VLM Proactiveness

Across desk-routine tasks, memory-augmented prediction improves over days, while FSM memory outperforms a flat list and continued reflection sustains improvement. Qualitative traces show the reflector adding rules, extending triggers, and changing activity phases.

B Per-task Results of VLM Proactiveness

  • Memory-augmented variants improve average F1 across T1–T6 from Day 1 to Day 5, while memory-free baselines achieve more limited overall performance.Memory helps especially on T2 (Milk→Person) and T3 (Cereal→Storage), where assistance depends on habitual pairing or routine progression; the gap is smaller on T1 and T6.Source: B Per-task Results of VLM Proactiveness · p. 18
  • FSM memory gains substantially over List Memory on storage tasks T3–T5, while its NO_ACT recall is lower and its NO_ACT precision higher, yielding comparable F1.Activity-phase context helps distinguish similar cues and choose when to put items away; the flat list encodes routine information in trigger strings.Source: B Per-task Results of VLM Proactiveness · p. 18

C.1 Effectiveness of Memory Evolution

  • Self-evolving memory improves from Day 1 to Day 5 and finishes with higher precision, recall, and F1 than frozen Day-1 memory or the memory-free baseline.The comparison averages over T1–T6; Figure 12 also tracks NO_ACT and first-fire delay.Source: §C.1 · p. 22, Figure 12 · p. 22
  • FSM memory reduces first-fire delay from roughly 5 seconds on Day 1 to about 1 second by Day 5, while frozen memory plateaus near 2 seconds.The variants maintain comparable NO_ACT F1, but self-evolving memory has lower NO_ACT recall, reflecting a proactiveness–false-alarm trade-off.Source: §C.1 · p. 22
  • Across one run, reflection adds assistance pairs and broadens triggers on Days 2–4, introduces a cleaning_break state on Day 4, and finishes broadening triggers on Day 5.Day 1 installs four states and two core pairs; on Day 5, actions and FSM topology remain fixed.Source: §C.1 · pp. 22–23

C.2 Reasoning Trace of Memory Update

  • The reflector can add a storage rule using an indirect cue: reaching for milk suggests an opportunity to put away the cereal bag while the person still holds it.The new pair is added to meal_preparation after the reflector reasons that the robot must not handle the bag while the person is holding it.Source: §C.2 · p. 23
  • When a person puts cereal directly into storage, the reflector extends the existing storage trigger with “Person finishes pouring cereal” rather than creating a new pair.This adds an earlier candidate cue when the desk-placement cue is absent.Source: §C.2 · p. 23
  • The reflector may create a separate cleaning_break phase for wet-wipe disposal, placing it in a working–cleaning loop to distinguish typing after cleaning from other typing.Organizing rules by phase helps interpret similar cues at different moments and limit mistimed assistance.Source: §C.2 · p. 23

C.3 Comparison of Memory Structures

By Day 5, FSM-structured PAM achieves higher average F1 across T1–T6 than List Memory. Its activity-phase context helps interpret cues that a flat trigger list can misread.

C.3 Comparison of Memory Structures

  • By Day 5, FSM-structured PAM achieves higher average F1 across T1–T6 than List Memory.The comparison illustrates how the two representations capture routine context; the FSM groups rules into states and transitions, while List Memory is a flat rule list.Source: §C.3 · p. 29
  • FSM Memory gives the predictor activity phases and transitions as context, whereas List Memory leaves it to infer routine context from trigger descriptions and observations.The predictor uses the FSM structure to interpret the scene and anticipate routine progress.Source: §C.3 · p. 29
  • During eating, FSM Memory outputs NO_ACT, while List Memory repeatedly triggers T2 (Milk→Person) from “holds the bowl,” despite milk already being poured.The phase context identifies eating as a state without a Milk→Person rule; the cue alone does not establish that more milk is needed.Source: §C.3 · p. 29, Figure 21 · p. 29
  • Associating milk delivery with meal_preparation can help limit repeated delivery after eating begins, but does not strictly prevent execution outside that phase.The List Memory trigger was broadened with “Person holds the bowl” without explicitly distinguishing meal preparation from eating.Source: §C.3 · p. 29

D.1 Evaluation on a Different Routine

On a different vitamins, water, and Coke routine, FSM Memory improves averaged task metrics over five days and reduces first-fire delay. The reported scenario-shift comparison also favors evolving over frozen source memory.

D.1 Evaluation on a Different Routine

  • The six tasks cover delivering vitamins, water, and Coke, returning the vitamin and water bottles to storage, and discarding the empty Coke can.The user takes vitamins with water, drinks Coke while working on a laptop, and then leaves.Source: §D.1 · p. 29
  • In the Scenario B-to-A adaptation comparison, evolving source memory adapts toward the from-scratch baseline, while frozen source memory underperforms.The A-to-B results are shown in Figure 6 in the main paper.Source: Figure 23 · p. 30
  • On the vitamins, water, and Coke routine, FSM Memory improves precision, recall, and F1 averaged over six tasks from Day 1 to Day 5, while reducing first-fire delay from roughly 17 s to 10 s.List Memory and memory-free baselines show limited improvement in this additional routine.Source: §D.1 · p. 30, Figure 22 · p. 30

D.2 Additional Results on Memory Adaptation

This section describes how list-based and FSM procedural memories are built and updated, and presents additional results on reverse vitamin-to-cereal adaptation. It also specifies the task-predictor and reflector prompts used with the memory variants.

D.2 Additional Results on Memory Adaptation

  • By Day 3, B–Evolving approaches Scratch– Evolving, while B–Frozen performs worse than the Zero-shot VLM, matching the A→B adaptation trend.The result is from the reverse vitamin-routine-to-cereal-routine evaluation.Source: §D.2 · p. 30

E Task Predictor Prompts

  • The task-predictor prompts use the final frame’s wall-clock time and five frames sampled at stride 2 from the 1 fps stream.The prompts retain their original 1 fps wording even though the supplied frames are 2 seconds apart.Source: E Task Predictor Prompts · p. 31

E.1 No Memory (baseline)

  • The no-memory predictor selects an immediately executable task from the current scene or returns task_index = -1 when no action should fire.The objective is based on the final, current frame, and actions on objects being held or touched are disallowed until the person sets them down.Source: §E.1 · p. 31

E.2 Ours (List Memory)

  • List Memory adds a procedural memory of flat natural-language trigger–action rules, which the predictor treats as a reference rather than a strict matcher.The predictor’s reasoning is instructed to reference the relevant memory rule.Source: §E.2 · p. 32

E.3 Ours (FSM Memory)

  • FSM Memory organizes assistance rules within routine-phase states and uses event-driven transitions, while asking the predictor to judge the memory holistically.The memory is a reference, not a strict matcher of trigger wording or state assignment.Source: §E.3 · p. 33

F Reflector Prompts

  • The reflector prompts cover building list memory from scratch and updating list or FSM memory at day’s end.They distinguish the flat production-rule list from the FSM’s phase-grouped rules and event-driven transitions.Source: F Reflector Prompts · p. 34

F.1 Ours (List Memory): build (no prior memory)

  • When building list memory, the reflector derives rules from observed human actions, mirrors only actions the person actually performed, and uses observable cues that assistance will not preempt.For motion-based triggers, the rule’s action must manipulate a different object from the one the person is handling.Source: §F.1 · pp. 34–35

F.2 Ours (List Memory): end-of-day update

  • For end-of-day list-memory updates, the reflector uses the current memory, video, summary, action history, and feedback to add, refine, extend, or remove rules under stated conditions.It defaults to adding uncovered take-over opportunities, extends a trigger when the human performed the rule’s action, and treats human feedback as authoritative.Source: §F.2 · pp. 35–36
  • List-memory updates output assistance rules as a trigger statement paired with a canonical pick-and-place action.The trigger describes the visible event that should cause the action to fire.Source: §F.2 · p. 36

F.3 Ours (FSM Memory): build (no prior memory)

  • The FSM is built from routine phases, with per-state assistance rules and event-driven transitions, using actions the person actually performed as the basis for robot take-over opportunities.Triggers must use observable cues and, when describing the person acting with an object, refer to a different object from the one the robot manipulates.Source: §F.3 · pp. 37–38

F.4 Ours (FSM Memory): end-of-day update

  • FSM memory updates use the day’s video, summary, current memory, and feedback to add or refine grounded rules; provided human feedback takes priority.The instructions favor adding uncovered take-over opportunities and removing rules only when clearly wrong, not merely because a trigger did not occur that day.Source: §F.4 · p. 39

Related papers

Loading 2610.21211v1…