PaperlayerView exact revision on arXiv

Source-linked AI summary

LogLook: Execution-Grounded Memory for Frontier-Model Robot Planners

Tianyi Jiang, Tianyao Tong, Zhenshan Bing

arXiv:2610.32348v1cs.RO

TL;DR

Manipulation planners need memory for actions and observations beyond the current image, but how frontier-model planners should remember an episode remains an open question. LogLook keeps an execution record while letting the model choose which frames and demonstration excerpts to inspect. With the benchmark’s motion planner, it matches note-only memory’s success using 43% fewer tokens and less than half as many images; direct control solves 93.5% of the RoboMME test set.

  • Problem

    Robot manipulation tasks require remembering prior actions and observations, while how a frontier-model planner should organize episode memory remains open.

    Source: §1 · p. 2

  • Method

    LogLook keeps an execution log and earlier notes, while the model selects changed frames near watched points and demonstration ranges to inspect.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    43% fewer tokens and less than half as many images match note-only memory’s success with the benchmark’s motion planner, while direct control solves 93.5% of the RoboMME test set.

    Source: §1 · pp. 2–3, §6 · p. 12

  • Takeaways & Limitations

    The results show that execution-grounded memory can match note-only success at lower token and image use, with fewer failures caused by memory or observation.

    Source: §6 · p. 12

    Evaluation uses one simulation benchmark and one planner model, with each configuration run once per episode.

    Source: §5 · p. 12

Abstract

from arXiv · show

Many manipulation tasks require a robot to remember what it has done and what it has seen. Frontier vision-language models can plan such tasks without task-specific training, but how such a planner should remember an episode is an open question. We argue that its memory must keep track of what the planner has done and learned, and must let the planner choose what to look at, since every frame costs tokens; a note rewritten at every step meets both poorly. We propose LogLook, whose execution-grounded memory lets the system keep the record, an execution log and the model's earlier notes, while the model decides where to look: frames are returned only where it expects a change, and demonstrations are skimmed before it inspects the parts it chooses. Measured with the benchmark's motion planner, LogLook matches the success of a note-only memory with 43% fewer tokens and less than half the images, and fails far fewer episodes through what it remembered or saw. With direct control, LogLook solves 93.5% of the RoboMME test set.

1 Introduction

RoboMME highlights memory as a bottleneck for manipulation, while frontier planners leave open how an episode should be remembered. LogLook assigns record keeping to the system and selective looking to the model, matching note-only success with reduced token and image use.

Motivation

  • RoboMME tasks require remembering prior actions and observations, and the best learned memory reaches about half the success of a π0.5 executor driven by oracle subgoals.Frontier vision-language planners can act without task-specific training, but how they should remember an episode remains open.Source: §1 · p. 2
  • A useful planner memory must preserve what was done and learned while letting the planner choose what to inspect, since rewrites can lose facts and frames cost tokens.Some facts, such as container swaps or whether a demonstrator pushed or carried a cube, are visible only briefly.Source: §1 · p. 2

Approach

  • LogLook keeps an execution log and all earlier notes, while the model selects watched image regions and demonstration excerpts; the design is task-independent and requires no training.Only recorded frames that change near watched points are returned after actions.Source: §1 · p. 2, Figure 2 · p. 3

Results

  • 43% fewer tokens and less than half as many images let LogLook match note-only memory’s success with the benchmark’s motion planner, while causing fewer failures from memory or observation.The note-only memory had 12 non-motion-planning failures versus 4 for LogLook; at least 11 of 12 versus one of 4 came from what the planner remembered or saw.Source: §1 · pp. 2–3
  • With direct control, the planner solves 93.5% of the RoboMME test set; with the learned executor, it comes close to oracle subgoals using a quarter of the tokens of a planner that replans after every action chunk.With the learned executor, swapping only the memory leaves success unchanged because executor failures dominate episodes where the memories disagree.Source: §1 · p. 2

2 Related Work

Prior work adds memory to learned robot policies and uses foundation models to plan robot behavior in several ways. LogLook instead studies how to organize memory for a frozen frontier planner, including the records it needs and what it should inspect.

Memory in robot manipulation policies

  • Robot-policy memories include visual-feature banks, perceptual-cognitive memory, state-change tokens, and VLM-selected keyframes; RoboMME [1] and RMBench [15] report gaps to oracle or human performance.These methods learn memory jointly with the policy; LogLook studies the complementary setting of a frozen frontier model.Source: §2 · p. 3

Foundation models as robot planners

  • Foundation-model robot planners have used skill scoring [16], textual feedback [17], code [18], spatial constraints or value maps [19] [20] [21], and VLM–VLA hierarchies [2] [14] [22].RoboICL [6] supplies demonstration context and interaction records, while Harness VLA [24] steers a frozen VLA with a memory-guided agent.Source: §2 · p. 3

Frontier models on RoboMME; memory and active perception

  • Unlike systems that decide when a planner is called or what is tracked for it [26], LogLook studies how the planner’s own memory should be organized.Language agents select video frames [31] [32], but robot planners also need records of their own actions and physical outcomes.Source: §2 · pp. 3–4

3 Method

LogLook studies what a frozen vision-language robot planner should remember and how the system and model should divide memory responsibilities. It combines a durable execution record with model-selected visual evidence.

3.1 Problem formulation

  • The planner receives the current observation, instruction, task note, and memory at each decision, then returns a note and action for an executor; earlier episode knowledge reaches it only through memory.The design objective is to succeed while showing the planner as few tokens and images as possible.Source: §3.1 · p. 4

3.2 Two requirements on the memory

  • A rewritten note can lose completed counts or earlier facts, while showing every frame is costly and fixed-stride sampling can miss brief events; the planner can choose what to inspect.Some facts appear only momentarily during an action or demonstration.Source: §3.2 · p. 4

3.3 Execution-grounded memory

  • The memory combines a system-kept execution log and earlier notes with frames and demonstration excerpts requested by the model.At each decision, the model receives the observation and memory, then returns a note and action.Source: §3.3 · p. 4

3.4 Keeping track of what was done

  • The system appends an entry for every executed action, including its command and measured gripper or end-effector state; the full log is shown at each decision.Earlier notes are also retained verbatim so dropped facts can be recovered; the note-only baseline shows only its last eight actions.Source: §3.4 · p. 5

3.5 Choosing what to look at

  • During actions, the system returns recorded frames only when pixels change near model-designated watch points; without a watch, no intermediate frames are returned.Demonstrations are first skimmed at every eighth frame, then the model can request ranges shown at every second frame, subject to a 24-frame cap.Source: §3.5 · p. 5

4 Experiments

The experiments compare execution-grounded memory with note-only memory, test the memory components, and evaluate the planner on RoboMME. They find similar success with lower memory cost in the controlled comparison and 93.5% success with direct control on the full test set.

4.1 Experimental setup

  • RoboMME has 16 tabletop tasks and 800 test episodes; nine tasks begin with demonstrations, and experiments use learned π0.5 and direct-control executors.Direct control is the main setting for measuring memory because it removes executor failures the planner cannot influence; success is determined by the environment.Source: §4.1 · pp. 5–7

4.2 Controlled memory comparison

  • 92.6% versus 91.4% success on 256 episodes accompanies 41k versus 71k tokens and 46 versus 114 images per episode.The comparison uses the benchmark motion planner; the figure also compares the learned-executor setting.Source: Figure 3 · p. 7, §4.2 · p. 7, Table 2 · p. 7
  • Among the 19 episodes solved only by the proposed execution-grounded memory, the note-only planner lost five to motion-planning failures and all but one of the other fourteen to memory or perception errors.These included five count or deposit omissions, three lost facts, and five misread demonstrations.Source: §4.2 · p. 8

4.3 What each component contributes

  • On 64 hard episodes, the full system has no failures other than motion-planning failures, versus two to four without a component and seven with note-only memory.Each individual component difference rests on only two to four episodes, so the authors treat these results as consistent with the failure analysis, not separate proof.Source: Table 3 · p. 8, §4.3 · p. 9
  • Expectation-gated frames reduce execution images from 83 to 19 per episode; skim-then-inspect halves demonstration images and places closer looks near demonstration events.Without skim-then-inspect, four of 36 demonstration episodes fail for non-motion-planning reasons; with it, none do.Source: §4.3 · p. 9, Figure 6 · p. 10

4.4 The full benchmark

  • 81.9% success with the full planner exceeds the per-chunk planner’s 75.1% and MemER’s 42.6% on 800 episodes.The full planner uses a quarter of the per-chunk planner’s tokens; its gain reflects the whole system, including call frequency and available actions.Source: Figure 7 · p. 10, §4.4 · p. 10, Table 5 · p. 11
  • Without task notes, the planner still solves 89.1% of test episodes 4–19, while non-motion-planning failures rise from 4 to 15.Completion criteria help on tasks whose success conditions cannot be seen in images, while most planner success does not depend on the notes.Source: §4.4 · pp. 11–12, Table 6 · p. 12
  • 93.5% of all 800 RoboMME test episodes are solved with direct control.The planner exceeds the π0.5 oracle-subgoal result in every suite; 35 of 52 failed episodes end in motion-planning failure.Source: §4.4 · p. 10, Figure 8 · p. 11

5 Limitations

The evaluation is limited to one simulation benchmark and one planner model, with each configuration run once per episode. The paper has not evaluated on a physical robot.

5 Limitations

  • Results cover one simulation benchmark and one planner model, with each configuration run once per episode.Repeated runs of the same episode do not always agree, so comparisons use paired episodes.Source: §5 · p. 12
  • The planner uses depth and camera calibration unavailable to the RoboMME learned baselines, so the paper also reports a learned-executor setting.That variant uses task notes containing conventions of the executor’s phrases.Source: §5 · p. 12
  • The system has not been evaluated on a physical robot, and GPT-6 Astra is a proprietary model accessed through an API.Source: §5 · p. 12

6 Conclusion

The paper presents execution-grounded memory, which preserves the record of actions and measurements while letting the model choose what to inspect. On RoboMME, it matches note-only memory success with lower image and token costs, while direct control solves 93.5% of the test set.

6 Conclusion

  • With the benchmark’s motion planner, execution-grounded memory reaches note-only memory’s success with well under half the image cost and far fewer memory- and observation-related failures.With direct control, the planner solves 93.5% of the test set.Source: §6 · p. 12
  • Execution-grounded memory keeps a record of executed and measured actions while the model chooses which frames and demonstration parts to inspect.Source: §6 · p. 12
  • Future work will evaluate the memory on a second benchmark, with other planner models, and on a physical robot.Source: §6 · p. 12

A Per-task results

The appendix presents per-task successes for memory-swap subsets, success by difficulty, and episodes where exactly one memory succeeds, with the other memory’s failure cause.

A Per-task results

  • Table 7 reports per-task successes on the memory-swap subsets for the learned executor and direct control.Its caption specifies test episodes 4–11 for the learned executor and 4–19 for direct control.Source: Table 7 · p. 15
  • Table 8 reports success by difficulty level for all 800 test episodes and for the memory swap with the motion planner.Source: Table 8 · p. 15
  • Table 9 lists memory-swap episodes where exactly one memory succeeds and gives the cause of the other memory’s failure.Failure causes were read from logs and checked against executed-step feedback.Source: Table 9 · p. 16

B Tasks, an example episode and failure cases

The appendix illustrates the RoboMME tasks, a successful direct-control episode, and examples of failure causes. The figures show what the planner must remember and selected frames or notes from episodes.

B Tasks, an example episode and failure cases

  • Figure 9 depicts all 16 RoboMME tasks, pairing what must be remembered with a frame from a successful direct-control episode.The line below each task names what the planner has to track.Source: Figure 9 · p. 17
  • Figure 10 follows a successful ButtonUnmaskSwap episode and shows selected frames alongside four of the nine notes.After six watched actions, the system returned 14, 7, 6, 7, 8, and 9 of the 79 recorded frames.Source: Figure 10 · p. 17
  • Figure 11 gives examples of failure causes, including note-only memory episodes solved by the memory and failures of the learned executor and direct-control planner.Source: Figure 11 · p. 18

C Implementation details

Table 10 presents the planner implementation settings, which are identical across tasks; the note-only memory differs from LogLook only in the final block.

C Implementation details

Loading 2610.32348v1…