OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation

1 Introduction

• A historical place must remain recoverable and actionable when its object has vanished or the robot views the workspace from a different pose.The same need arises for identifying an earlier instance, recovering a pre-motion pose, or recalling shuffled-container contents.Source: §1 · p. 1
• Prior history-aware policies and sampled-frame VLMs address historical context, but more frames alone do not provide a persistent metric reference frame for a new viewpoint.Retrieval-based methods such as MemER select relevant keyframes for hierarchical robot control.Source: §1 · p. 1
• OCC4M combines world-frame object tracks and explicit temporal, motion, and containment relations with a VLM that selects targets and frozen low-level skills that execute grounded cues.The system focuses on binding historical references to actionable geometry rather than introducing a new detector, scene mapper, or low-level policy.Source: §1 · p. 1
• OCC4M distinguishes a fixed historical location from a persistent object's live location, which can update during execution.A vanished place stays fixed under ego-motion, while a selected object's target can follow its track.Source: §1 · p. 2
• The system is evaluated through record–replay comparisons, a viewpoint control, seven simulation conditions, and a paired real-world memory study.The evaluation also includes oracle diagnostics and a hardware perturbation showing live target updates after one planning call.Source: §1 · p. 2

2.1 From pixels to a shared world frame

• Detections are converted to world points by intersecting calibrated camera rays with the support plane, and current cues reproject stored points through the current camera pose.Ego-motion changes the image projection, not the stored historical place; the implementation assumes known support geometry.Source: §2.1 · p. 2

2.2 Persistent object tracks

• The memory keeps multiple same-class instances as separate tracks, matches detections by planar world distance, and retains unmatched tracks at their last detected poses.A containment-inferred location remains distinct from an independently detected object pose.Source: §2.2 · p. 2
• Table 1 identifies T1c as combining a new camera pose with a current-frame grounding requirement.This condition tests whether the target remains actionable after viewpoint transfer.Source: Table 1 · p. 3
• Track histories encode first-observation order, motion events with pre-motion and current poses, and directed containment links that infer a disappeared object's location from its cover.Given detections, camera poses, and fixed thresholds, these updates are deterministic.Source: §2.2 · p. 3

2.3 Memory-to-action interface

• The VLM reads a typed-graph serialization once per episode, selects object identities and an ordered skill plan, and memory re-grounds object-bound targets to updated tracks during execution.Historical-location targets retain their recorded coordinates rather than following later motion of the originally observed object.Source: §2.3 · p. 3
• The frozen low-level policy receives current images, joint state, a skill word, and a grounded target cue, but not the task instruction, memory, or prior frames.The controller reprojects the target into the current shoulder image for execution.Source: §2.3 · p. 3

3.1 Task suite and evaluation protocol

• Seven simulation conditions use the same 50 recorded evaluation seeds, replayed with identical RGB histories and goals across methods.Memory success measures complete identity and target decisions before motion, while end-to-end success uses task-specific criteria and is not nested within memory success.Source: §3.1 · pp. 3–4
• The principal FrameSamp baseline reasons over sampled frames, receives the complete history at K = 32, and may ground targets in any supplied historical frame.The simulation executor is held fixed, but OCC4M and FrameSamp use different VLM backbones and input representations.Source: §3.1 · p. 4

3.2 Viewpoint-transfer design

• T1b requires current-frame grounding without camera motion, while T1c adds a 0.30 m base translation after the target disappears.Together the controls distinguish failure from the output constraint alone from failure after viewpoint change.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 Real-world evaluation

• The Franka study uses a fixed camera and normalized image-plane coordinates, so it does not test world-frame viewpoint invariance.Its 20 randomized episodes pair memory comparisons on identical histories; only OCC4M is physically executed.Source: §3.3 · p. 4
• The perturbation demonstration moves a selected target cup about 20 cm during approach and re-grounds its cue to the live track every 0.8 s action-chunk boundary.The identity is selected before execution, with no new VLM query described.Source: §3.3 · p. 5

4.1 Main comparison

• 96.6% memory success and 88.9% end-to-end success for OCC4M compare with 54.6% and 57.7% for full-history FrameSamp across seven conditions.Table 2 reports means across the seven task-condition rates, with 50 evaluation episodes per condition and the same frozen executor for learned-action rows.Source: Table 2 · p. 5, §4.1 · p. 5
• FrameSamp’s full-history budget raises mean memory success from 52.3% to 54.6%, but its best observed mean success is 9.3% versus 87.3% for OCC4M in the separate trained-baseline comparison.Denser sampling improves T4 memory success from 54% to 74% but does not close gaps on T3, T6, or T1c.Source: §4.1 · p. 5

4.2 Ego-motion-invariant recall

• Under viewpoint transfer, OCC4M reaches 100% memory and 98% end-to-end success, while FrameSamp reaches 6% on both measures.OCC4M is correct on all 50 T1c episodes; median error is 1.2 cm for OCC4M and 28.8 cm for FrameSamp under the 30 cm displacement.Source: §4.2 · p. 6, Table 3 · p. 6, Figure 2 · p. 6

4.3 Error decomposition

• Oracle actions add 29/350 successes, or 8.3 percentage points, raising end-to-end success to 97.1%; the authors identify execution as an important remaining bottleneck.These variants use separate rollouts and, where applicable, new VLM calls, so they are aggregate diagnostics rather than perfectly paired causal interventions.Source: §4.3 · p. 6
• On T6, OCC4M has 100% memory success but 62% end-to-end success, while oracle actions reach 100%.All 19 OCC4M failures followed usable memory/plan decisions; FrameSamp often returned incomplete multi-step plans.Source: §4.3 · p. 6

4.4 Real-world evaluation

• 85% joint memory accuracy on the 20 real-world episodes for OCC4M compares with 30%, 25%, and 20% for FrameSamp at K = 16, K = 32, and all frames.The gap centers on shuffled-containment recall; all methods perform well on historical-location recall.Source: Table 4 · p. 7, §4.4 · p. 7
• OCC4M completes 9/20 full two-stage Franka episodes (45%), despite 85% joint memory accuracy.The physical executor succeeds on 15/20 first subtasks and 11/20 second subtasks; the executor was trained from 54 pick demonstrations.Source: §4.4 · p. 7
• After a target cup moves during execution, OCC4M updates its location at 10.6 Hz without re-querying the VLM and grasps it at the displaced location.The cue is refreshed at action-chunk boundaries, implying approximately 0.5 s expected retargeting latency under a uniform chunk-phase assumption.Source: §4.4 · p. 7, Figure 4 · p. 7

Historical retrieval and actionable spatial memory

• A historical place remains fixed across ego-motion, whereas a selected object’s target follows its live track and can be refreshed without repeating task-level reasoning.In the perturbation demonstration, OCC4M updates tracks at 10.6 Hz and grasps the displaced cup without a new VLM call; reported query timings are deployment-dependent.Source: §5 · p. 8
• The evaluation does not isolate memory structure as the sole source of the observed gap because the compared systems use different VLMs and input representations, and raw history augmented with explicit geometry is untested.Source: §5 · p. 8

Physical grounding and execution

• OCC4M’s physical study reports 85% joint memory accuracy and 45% end-to-end completion, while only OCC4M is executed physically and its fixed-camera setup does not test mobile world-frame transfer.The single perturbation demonstration shows live grounding of an already selected identity, not general disturbance robustness or high-level plan repair.Source: §5 · p. 8

Limitations

• Evaluation is limited to small structured scenes and scripted histories; known support geometry, fixed association thresholds, and uncalibrated point estimates constrain robustness under clutter, long occlusions, and ambiguous associations.The authors identify matched-backbone and component ablations, uncertainty-aware association, and real-world mobile viewpoint transfer as next tests.Source: §5 · p. 8

• Across seven simulation conditions, OCC4M achieves 96.6% memory and 88.9% end-to-end success, versus 54.6% and 57.7% for a full-history VLM baseline with the same executor.Source: §6 · p. 8
• Viewpoint controls show the value of preserving a world-frame place across ego-motion, while physical experiments provide initial evidence for relational recall and live target grounding.Source: §6 · p. 8
• The results support persistent object-centric state as a practical basis for embodied spatiotemporal reasoning and identify execution and association as priorities for more reliable deployment.Source: §6 · p. 8

Unified executor

• Both OCC4M and the shared-executor FrameSamp baseline use one frozen π0.5 LoRA checkpoint fine-tuned for 5,000 steps on 1,100 atomic demonstrations.The demonstrations comprise 700 pick and 400 place examples; the policy receives current shoulder/wrist RGB, joint state, one atomic skill word, and one target cue.Source: A Implementation Details · p. 10

Tracking and control

• Detections are associated within semantic classes by Hungarian assignment on planar world distance, gated at 12 cm around a constant-velocity prediction; motion and containment events use fixed geometric rules.Motion onset is recorded beyond 5 cm from rest, and an object disappearing beneath a cover within 11 cm is linked to that cover until reappearance; T6 does not establish cross-appearance re-identification.Source: A Implementation Details · p. 10

Real-world perception timing

• The perturbation tracker performs 382 updates over 36.0 s, or 10.6 updates/s, while estimated cue-refresh delays are about 0.5 s on average and 0.9 s nominally at most under the stated schedule.These are scheduling estimates, not measured latency quantiles or hard real-time guarantees.Source: A Implementation Details · p. 10

Memory-Conditioned Policy Baseline

• Across tested FrameSamp+Modul settings, mean success remains below 10%, compared with 87.3% for OCC4M on the same six conditions, excluding T1c.The best observed mean success is 9.3% with 400 demonstrations per task at 20,000 steps.Source: B Memory-Conditioned Policy Baseline · p. 10
• With 250 demonstrations per task, FrameSamp+Modul reaches 5.0% mean end-to-end success at 20,000 steps and does not improve with further training.The policy is evaluated every 10,000 steps for up to 40,000 gradient steps.Source: B Memory-Conditioned Policy Baseline · p. 10
• The sweep does not establish why success remains low: optimizer settings, architecture adaptation, and demonstration quality remain possible explanations, and the score excludes T1c.Source: B Memory-Conditioned Policy Baseline · p. 10

C VLM Memory-Baseline Evaluation

• Memory success requires maximum target error below 5 cm and satisfaction of task-specific identity constraints.The maximum covers required pick and place targets, except pick-only tasks score only pick targets.Source: C VLM Memory-Baseline Evaluation · p. 11
• Table 6 presents task-specific definitions of the executor-independent memory metric.For T6, instances are matched one-to-one within original color classes, predicted counts must match ground-truth counts, and fixed bin locations are excluded from the memory metric.Source: Table 6 · p. 11, C VLM Memory-Baseline Evaluation · p. 11
• Predicted image points are back-projected into world coordinates using the supplied frame’s calibration and target-height support plane.The VLM receives non-privileged 224 × 224 RGB frames, while camera calibration and base pose are withheld from it.Source: C VLM Memory-Baseline Evaluation · p. 11
• Frame constraints vary by task: T4 grounds the pick in the final frame, T1c the place in the post-motion frame, and T5 the place in the original Table-A frame.T1b also requires the final frame for the place prediction while keeping the camera fixed; when fewer than K observations exist, all available frames are supplied once.Source: C VLM Memory-Baseline Evaluation · p. 11
• Each transfer prediction includes pick and place points, a pick label, and frame identifiers; malformed or empty outputs count as memory failures without re-prompting.Predictions use one cycle per required object transfer.Source: C VLM Memory-Baseline Evaluation · p. 11

D Qualitative Task Sequences

• Place the red cube where the green cube was, then place blue and red cubes at the first and second marker locations, respectively.These sequences require using a prior cube location and the temporal order in which markers appeared.Source: Figure 5 · p. 12, Figure 7 · p. 12
• After the green cube disappears and the base moves 0.30 m, place the red cube at the green cube’s remembered location from the new viewpoint.The figure marks OCC4M’s prediction in yellow, FrameSamp’s in pink, and the ground-truth target in green.Source: Figure 6 · p. 12
• Pick the second mover and place it at the first mover’s pre-motion location.Source: Figure 8 · p. 12
• Pick the grey cover containing the red cube after shuffling; another sequence places a query-color cube at the remembered sign location before sorting cubes by original color.Source: Figure 9 · p. 13, Figure 10 · p. 13, Figure 11 · p. 13