Source-linked AI summary
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
Yansong Shi, Jiange Yang, Xijie Yang, Shaowei Zhang, Yuhan Zhu, Tao Lu, Limin Wang
TL;DR
Reliable manipulation can depend on task states hidden from current observations, and task goals may not specify execution progress. The paper introduces HIDE to benchmark these requirements and SEEK to maintain relevant memory. Evaluations show improvements across hidden-state tasks, with the combined mechanisms achieving the strongest overall performance.
Problem
Manipulation policies often rely on current observations or limited temporal context, although relevant task information may be hidden and task objectives may not specify execution state.
Source: §1 · p. 1Method
The paper introduces HIDE, a 15-task benchmark for three hidden-state requirements, and SEEK, which maintains recent context, persistent historical references, and execution progress.
Source: §1 · p. 2Results
Takeaways & Limitations
Memory mechanisms have distinct capability profiles, benefiting some hidden-state requirements while potentially degrading others.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce $HIDE$, a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose $SEEK$, a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.
1 INTRODUCTION
The paper frames manipulation memory around hidden task states that must be recovered from interaction history, then introduces HIDE to evaluate these requirements and SEEK to address them. SEEK improves performance in simulation and real-world experiments, with its combined mechanisms achieving the strongest overall performance.
1 INTRODUCTION
- Manipulation policies often rely on current observations or limited temporal context, although decision-relevant information may be hidden and recoverable from interaction history.Such histories can produce visually similar observations that require different actions.Source: §1 · pp. 1–2
- A task objective alone may not specify execution state: an instruction to repeat an operation twice does not say how many executions are complete.The paper argues that policies must maintain relevant internal states rather than relying only on externally specified goals.Source: §1 · p. 1
- HIDE is a 15-task benchmark covering repetition counting, historical-state recall, and execution-progress tracking, with randomized configurations and appearance variations.It also provides automated demonstrations and structured hidden-state annotations for memory analysis.Source: §1 · p. 2
- Existing memory policies retain history through recurrent states, memory banks, or retrieval, but retaining history does not ensure recovery of the hidden state a decision requires.Prior work also reports task-dependent memory effectiveness, motivating designs matched to hidden-state requirements.Source: §1 · p. 2
- SEEK improves manipulation performance in simulation and real-world experiments, and its combined mechanisms achieve the strongest overall performance.Its individual mechanisms benefit some hidden-state requirements while potentially degrading others.Source: §1 · p. 2
2 RELATED WORK
Prior manipulation benchmarks emphasize task completion and generalization, while memory-oriented work explores historical representations and retrieval. This paper positions HIDE around hidden variables needed for correct decisions and SEEK around complementary mechanisms evaluated across those requirements.
2.1 MEMORY-DEPENDENT ROBOTIC BENCHMARKS
- HIDE organizes tasks by hidden variables needed for correct decisions and constructs points where observations are insufficient but interaction history provides necessary evidence.This complements benchmarks focused primarily on task completion, generalization, task length, scene complexity, or retained-context amount.Source: §2.1 · p. 3
2.2 MEMORY-AUGMENTED ROBOTIC POLICIES
- Memory-augmented policies use temporal windows, recurrent states, explicit memory, or retrieval, with trade-offs in temporal coverage, information retention, and online maintenance.Examples include trajectory-based object dynamics, compressed multi-frame context, visual memory banks, and retrieval for long-horizon control.Source: §2.2 · p. 3
- Retaining observations alone does not guarantee recovery of decision-relevant hidden states, so SEEK combines recent context, persistent references, and execution progress.The paper evaluates these mechanisms' benefits, limitations, and interactions through controlled cross-category experiments.Source: §2.2 · p. 3
3 HIDE
HIDE frames manipulation memory around hidden task states that history can reveal when current observations are insufficient. It groups tasks by the memory required for correct decisions and builds varied, staged benchmark sequences on RLBench.
3.1 HIDDEN-STATE-DEPENDENT MANIPULATION
- HIDE models a hidden task state alongside observable physical state because histories can require different actions despite visually equivalent observations.The hidden state represents historical information required for optimal decision making; long horizons or occlusion alone do not define memory dependence.Source: §3.1 · pp. 3–4
3.2 TASK CATEGORIES AND DESIGN
- HIDE groups tasks by primary memory need: repetition counting, historical-state recall, or execution-progress tracking, each tested at history-dependent decision points.Tasks may involve additional memory demands, while occlusion, visual similarity, and repeated interactions create history dependence rather than separate categories.Source: §3.2 · p. 4
- Repetition-counting tasks require tracking completed operations to decide whether to continue, stop, or switch when repeated actions look similar.The category tests accurate counting throughout interaction, not merely reproducing a recurring action pattern.Source: §3.2 · p. 4
- Historical-state recall tasks require acting on previously observed object or scene information that is no longer directly accessible at the decision point.Occlusion, scene changes, or visually confusable alternatives can make current-scene recognition insufficient.Source: §3.2 · p. 4
- Execution-progress tasks use interaction history to select the next substep when intermediate observations do not reveal which steps are complete.The category targets avoiding unnecessary repetition, skipped steps, or premature termination under observational ambiguity.Source: §3.2 · p. 4
3.3 BENCHMARK CONSTRUCTION
- HIDE extends RLBench with randomized object poses and appearances, longer memory-dependent sequences, and expert demonstrations segmented into task stages.Difficulty can vary with repetition counts, distractor number and similarity, delays after informative observations, and the number of manipulation stages; stage labels are generated automatically.Source: §3.3 · pp. 4–5
4 SEEK
SEEK combines recent interaction context, retrieved historical evidence, and an explicit progress state in a compact memory input. Its memories condition a coarse-to-fine policy, with encoded interactions written for later use and progress updated online during behavior-cloning training.
4 SEEK
- SEEK combines recent context, a retrieved long-range reference, and execution progress to address distinct history-dependent decisions under partial observability.The three components are Windowed Context Memory (WCM), Persistent Anchor Memory (PAM), and Stage-Counter Memory (SCM).Source: §4 · p. 5
4.1 HIDDEN-STATE MEMORY MODELING
- WCM retains the latest K interactions, PAM retrieves eligible historical evidence outside that recent window, and SCM tracks progress with a counter advanced at predicted stage boundaries.The counter remains stable while an action unfolds; unlike WCM and PAM, it explicitly represents how far execution has progressed.Source: §4.1 · pp. 5–6
4.2 MEMORY ENCODING AND RETRIEVAL
- The shared memory encoder associates scene features with a predicted manipulation target, retaining spatial feature maps so retrieval provides localized historical information.After action prediction, each encoded interaction becomes available to subsequent decisions; current visual features attend to selected memories, with at most K recent entries, one anchor, and one stage representation per view.Source: §4.2 · p. 6
4.3 MEMORY-AUGMENTED MANIPULATION POLICY
- SEEK conditions a coarse-to-fine, multi-view policy on memory for workspace target selection before the fine branch refines the interaction using local geometry.The coarse branch predicts translation heatmaps; the fine branch predicts position, rotation, and gripper state, while stage-transition predictions update SCM and all memories reset between episodes.Source: §4.3 · p. 6
4.4 TRAINING OBJECTIVE
- SEEK uses behavior cloning on expert demonstrations, with action losses and a stage-boundary loss training the progress predictor.At inference, memory and the counter update online from policy predictions, and retrieval uses only preceding observations.Source: §4.4 · p. 7
5 EXPERIMENTS
The experiments evaluate SEEK on HIDE and beyond it, finding gains over existing policies on hidden-state tasks and strong or competitive performance in other settings.
5.1 EXPERIMENTAL SETUP
- Evaluation covers 15 HIDE tasks, 18 RLBench tasks, The Colosseum perturbations, and four real-robot tasks, using 100 training demonstrations and 25 test episodes per HIDE task.A single policy is jointly trained across all 15 HIDE tasks; SAM2Act is the backbone baseline.Source: §5.1 · p. 7
5.2 Q1–Q2: HIDDEN-STATE MEMORY GAP AND SEEK PERFORMANCE
- Baseline performance varies by hidden-state requirement: SAM2Act+ leads RC and EPT, while GR00T-N1.7 leads HSR at 49.6%.SAM2Act+ averages 51.2%; MME reaches 35.2% on HSR but 2.4% on RC.Source: §5.2 · p. 7
- SEEK reaches 62.9% average success on HIDE, 11.7 percentage points above SAM2Act+, and leads RC (61.6%), HSR (59.2%), and EPT (68.0%).Its category-wise gains over the strongest baselines are 15.2, 9.6, and 7.2 points, respectively.Source: §5.2 · p. 8
5.3 Q3: HOW DO DIFFERENT MEMORY COMPONENTS CONTRIBUTE?
- 62.9% average success with visual-feature memory versus 42.5% without it, with gains in all categories and the largest gain on RC (+30.4 percentage points).The content variants share encoding dimensionality and memory architecture.Source: §5.3 · p. 8, Table 2 · p. 8
- 62.9% average success at memory length 6 versus 46.7% at length 0; RC initially falls from 46.4% to 44.2% at length 2 before rising to 61.6%.Length 6 gives the best overall and category scores; EPT increases from 67.2% at length 4 to 68.0% at length 6.Source: §5.3 · pp. 8–9
- Combining SCM, WCM, and PAM leads all three categories, while their largest progressive gains occur on RC with SCM (+14.0 points), EPT with WCM (+12.0), and HSR with PAM (+11.6).Among individual variants, SCM leads RC, PAM leads HSR, and WCM leads EPT.Source: Figure 4 · p. 8, §5.3 · p. 9
5.4 Q4: DOES THE METHOD REMAIN EFFECTIVE BEYOND HIDE?
- SEEK achieves 84.7% average success on RLBench versus 84.1% for the SAM2Act reproduction, and 68.6% clean and 61.9% perturbed success on The Colosseum.The SAM2Act reproduction scores 68.4% clean and 61.5% averaged over perturbations; relative drops are 9.8% versus 10.1%.Source: Table 3 · p. 8, §5.4 · p. 9
- SEEK achieves 89% average success across four randomized real-world tasks, compared with 47% for SAM2Act+ and 13% for π0.5.The tasks are repeated button pressing, cup stacking, desk cleaning, and searching for a hidden white piece.Source: §5.4 · p. 9, Table 4 · p. 9
6 CONCLUSION
The conclusion presents HIDE and SEEK as tools for studying and addressing hidden task states in manipulation, while identifying richer settings as future work.
6 CONCLUSION
- HIDE organizes hidden-state manipulation around repetition counting, historical-state recall, and execution-progress tracking; SEEK maintains task-relevant information over interaction history.The paper reports improvements across these categories and strong performance on standard manipulation, perturbations, and real-world execution.Source: §6 · p. 9
- The authors identify HIDE’s focus on controlled, interpretable hidden states as a scope boundary and call for richer settings, broader partial observability, and more general memory mechanisms.Source: §6 · p. 9
- The authors report using generative AI for literature search, experiment scripting, initial drafting, language polishing, and formatting, with human review of assisted work.They state that AI was not used for research ideation, experimental design, implementation, data construction or analysis, or final writing and revision.Source: AI USE STATEMENT · p. 10
- The work reports no human subjects or personal data collection and describes real-robot evaluations in supervised indoor laboratory settings.Source: ETHICS STATEMENT · p. 10
A.1 SIMULATION TASKS
HIDE’s simulation tasks are grouped by whether correct manipulation depends on counting repetitions, recalling past scene information, or tracking execution progress.
A.1 SIMULATION TASKS
- HIDE contains 15 tasks in three categories, with five tasks each; Table 5 reports language templates, frame and keyframe averages, variations, and variation types.Source: §A.1 · p. 14, Table 5 · p. 14
I. Repetition Counting
- Repetition-counting tasks require exact requested counts, including button presses, block or cup stacking, and placing blocks into a container.Source: §I · pp. 14–15
II. Historical-State Recall
- Historical-state recall tasks require acting on earlier information, such as a measured weight, a prior object arrangement, a completed substep, or which drawer was initially open.Source: §II · pp. 15–16
III. Execution-Progress Tracking
- Execution-progress tasks require using history to determine completed substeps and what comes next while searching for or relocating objects.Examples include searching drawers or a shoe box, retrieving a cabinet cup, finding a hidden chip, and swapping rings.Source: §III · pp. 16–17
A.2 REAL-WORLD TASKS
The real-world evaluation uses four manipulation tasks with randomized initial configurations and reports their task and trajectory statistics.
A.2 REAL-WORLD TASKS
- The four real-world tasks are repeated button pressing, cup stacking, desk cleaning, and searching for a hidden white piece.Source: §A.2 · p. 17
- Visually similar observations can correspond to different latent states and require different actions, challenging policies that rely primarily on the current observation.Source: §A.2 · p. 17
- Table 6 records language templates, trajectory statistics, and variation types for the four tasks.Source: Table 6 · p. 17
I. Real-world Tasks
The physical tasks require the robot to track repetition counts, task stages, or previously inspected locations. Their success criteria specify when the robot must stop or finish.
I. Real-world Tasks
- Button-pressing succeeds only when the target colored button is pressed exactly the requested number of times, with no additional press.Similar observations before and after presses require tracking the completed count.Source: §I · p. 18
- Cup-stacking succeeds when the requested number of cups is stacked on the middle cup and the robot is no longer holding a cup.Similar intermediate configurations can represent different completion stages.Source: §I · p. 18
- Desk cleaning requires removing rubbish first, then cleaning all required desk regions while tracking completed substeps and earlier scene information.Removing the rubbish reveals the area that needs cleaning.Source: §I · p. 18
- Block inspection proceeds sequentially until the hidden white piece is found, requiring memory of which blocks have already been checked.Success also requires completing the specified final action.Source: §I · p. 18
A.3 TASK VISUALIZATION
The visualization highlights cases where similar observations correspond to different hidden task states and therefore require different next actions. The accompanying sections specify HIDE, the simulation setup, and evaluation metrics.
A.3 TASK VISUALIZATION
- Figures 5–7 illustrate HIDE’s repetition-counting, historical-state-recall, and execution-progress-tracking tasks, including visually similar observations tied to different hidden states and actions.The current observation alone can be insufficient; recovering task-relevant interaction history is necessary to determine the next action.Source: §A.3 · pp. 18–19, Figure 5 · p. 19, Figure 6 · p. 20
B.1 BENCHMARKS AND DATA
- HIDE extends RLBench with 15 tasks across three five-task categories, using 100 training demonstrations and 25 held-out test episodes per task.HIDE is the principal benchmark for comparisons and component ablations; RLBench evaluates general manipulation and The Colosseum evaluates robustness in clean and perturbed scenes.Source: §B.1 · p. 22
B.2 SIMULATION AND OBSERVATIONS
- Simulation uses a 7-DoF Franka Panda with four-camera RGB-D observations, while SEEK is trained in two stages that separate visual adaptation from learning to use interaction history.The policy predicts keyframe end-effector pose and gripper actions executed by an OMPL-based planner; the comparison includes VLA, 3D manipulation, and memory-based policy families, with SAM2Act as the memory-backbone reference.Source: §B.2 · p. 22
B.3 EVALUATION METRICS
- Task success rate is the primary metric, reported per task, by five-task category, and as an unweighted average over HIDE’s 15 tasks.The Colosseum reports clean and perturbed success rates separately.Source: §B.3 · p. 22
B.4 REAL-ROBOT EVALUATION
The real-robot evaluation tests four task types against two comparison methods, while the training-table notes specify SEEK’s stages and reported checkpoints.
B.4 REAL-ROBOT EVALUATION
- SEEK is evaluated against SAM2Act+ and π0.5 on four real-robot tasks, using 25 test trials per task and separately trained policies.The tasks are button pressing, cup stacking, desk cleaning, and lifting blocks to find a white piece; each task has 50 real-robot demonstrations and predefined variations represented uniformly in collection and evaluation.Source: §B.4 · p. 22
- Table 7 distinguishes Stage 1 memory-free policy learning from Stage 2 memory-augmented training initialized from the Stage 1 checkpoint.The table’s batch sizes are global across all GPUs.Source: Table 7 · p. 23
- The reported model uses the Stage 1 epoch-20 and Stage 2 epoch-10 checkpoints, while retaining cosine schedules configured for 40 and 20 epochs.Scheduled update counts include updates skipped by mixed-precision loss scaling.Source: §B.4 · p. 23
C DETAILED RESULTS
The detailed-results material identifies comparison tables for RLBench and The Colosseum and shows the real-world scene and task setups. The supplied passages describe table scope but do not provide numerical outcomes.
C DETAILED RESULTS
- Figures 8–9 show the real-world scene and four task setups: repeated button pressing, cup stacking, desk cleaning, and lifting blocks to search for a white piece.The captions identify the setup but do not report task outcomes.Source: Figure 9 · p. 24, Figure 8 · p. 24
- Table 8 compares success rates across 18 RLBench tasks and identifies the last two rows as the method without and with memory.Its caption explains that some SAM2Act results are from the original paper and others use officially released weights under the authors’ evaluation protocol.Source: C DETAILED RESULTS · p. 23, Table 8 · p. 25
- Table 9 compares success rates across clean scenes and 12 individual Colosseum perturbations, with All-Mixed excluded from the perturbation average.Relative changes are measured against clean results on the corresponding task subsets; legacy RVT entries are excluded from ranking.Source: C DETAILED RESULTS · p. 23, Table 9 · p. 25