Source-linked AI summary
BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames
Max Sobol Mark, Jacky Liang, Maria Attarian, Chuyuan Fu, Debidatta Dwibedi, Dhruv Shah, Aviral Kumar
TL;DR
Many robot tasks require memory, yet naïve history-conditioned imitation policies can exploit spurious correlations because training cannot cover exponentially many possible histories. BPP compresses histories into task-relevant semantic keyframes detected with a vision-language model, and it outperforms comparison methods across seven memory-dependent tasks, while deployment remains constrained by VLM latency and keyframe-detection accuracy.
Problem
History-conditioned imitation policies can latch onto spurious correlations because the space of possible observation histories grows exponentially with horizon and is poorly covered by training data.
Method
BPP uses an off-the-shelf vision-language model to detect task-relevant semantic keyframes and conditions policies on these compact events instead of raw observation histories.
Results
Across three simulation and four real-world tasks, BPP outperforms state-of-the-art history-conditioned approaches by up to 70%.
Takeaways & Limitations
Conditioning on semantic events rather than raw temporal sequences reduces distribution shift between training and deployment while supporting robust progress tracking.
Takeaways & Limitations
Real-time deployment is constrained by external VLM query latency, and systematic keyframe-detection errors can trigger premature state transitions.
Abstract
from arXiv · showhide
Many robot tasks require attending to the history of past observations. For example, finding an item in a room requires remembering which places have already been searched. However, the best-performing robot policies typically condition only on the current observation, limiting their applicability to such tasks. Naively conditioning on past observations often fails due to spurious correlations: policies latch onto incidental features of training histories that do not generalize to out-of-distribution trajectories upon deployment. We analyze why policies latch onto these spurious correlations and find that this problem stems from limited coverage over the space of possible histories during training, which grows exponentially with horizon. Existing regularization techniques provide inconsistent benefits across tasks, as they do not fundamentally address this coverage problem. Motivated by these findings, we propose Big Picture Policies (BPP), an approach that conditions on a minimal set of meaningful keyframes detected by a vision-language model. By projecting diverse rollouts onto a compact set of task-relevant events, BPP substantially reduces distribution shift between training and deployment, without sacrificing expressivity. We evaluate BPP on four challenging real-world manipulation tasks and three simulation tasks, all requiring history conditioning. BPP achieves 70% higher success rates than the best comparison on real-world evaluations. Videos are available at https://bigpicturepolicies.github.io/
1. Introduction
History is necessary for many robot tasks, but naïve history conditioning often fails because training data poorly covers the exponentially growing space of possible histories. BPP addresses this by conditioning on compact, task-relevant semantic keyframes, improving generalization and real-world performance.
- Motivation: Many manipulation tasks require remembering past events because the current observation alone cannot determine the correct action.Examples include searching cluttered scenes, counting ingredients, and executing multi-stage procedures.
- Problem: Naïvely adding observation history often hurts performance because policies learn spurious correlations in training histories that fail to generalize during deployment.These failures can occur even on simple tasks.
- Problem: The possible history space grows exponentially with horizon, making complete training coverage impossible under near-expert demonstration collection.Regularization, auxiliary objectives, and architectural constraints can help inconsistently but do not resolve the underlying coverage problem.
- Approach: BPP replaces raw histories with a small set of semantic keyframes representing behaviorally salient events such as grasping, subgoal completion, or failure.This reduces the effective input-history space while preserving task-relevant information for action prediction.
- Results: Across four real-world and three simulated tasks, BPP substantially outperforms strong history-conditioned and memoryless baselines, achieving up to 70% higher real-world success rates.The method is also reported to improve data efficiency and reduce training time.
2. Related Work
Prior robot policies commonly avoid long histories because they introduce spurious correlations and causal confusion. Related approaches mitigate these problems through reweighting, auxiliary objectives, traces, or information bottlenecks, while BPP uses VLMs to filter histories into task-relevant keyframes.
- History in robot learning policies: Most high-performing robot policies condition only on the current observation, while memory is mainly studied for tasks involving partial observability or required history.History conditioning is less common in imitation learning because it often harms performance.
- Addressing spurious correlations: Observation histories can create spurious correlations and causal confusion under rollout distribution shift, including implicit leakage of past actions.Extended sequences can also encourage policies to exploit irrelevant temporal patterns even without explicit action histories.
- Addressing spurious correlations: Prior remedies include loss reweighting on action-change keyframes, keyframe discovery for planning, information bottlenecks, and auxiliary past-token prediction.The paper reports that past-token prediction with test-time selection is effective only under certain task conditions.
- Related approaches: Some methods avoid raw history conditioning by augmenting observations with visual traces, while other systems use pretrained vision-language models for action prediction, planning, rewards, or value estimation.These approaches differ from BPP’s use of a VLM to filter observation histories for learning.
- Novelty: The paper states that prior work had not used VLMs to filter observation histories to improve learning signals.
3. What Makes History-Conditioned Imitation Learning Particularly Challenging?
History-conditioned imitation is difficult because long observation histories expose policies to spurious correlations under rollout distribution shift. Action chunking and jointly trained encoders reduce catastrophic failures, but limited history coverage remains the fundamental bottleneck.
- Action chunking: 2.9× versus 7.2× error increase shows that longer action chunks produce history features that generalize better to policy rollouts.Short action chunks substantially hurt naïve history-conditioned policies, while oracle policies are affected little.
- Policy architecture: Jointly training the image encoder across all timesteps yields better history conditioning than freezing an encoder trained with short contexts.The result suggests that encoder inflexibility contributes to poor performance on extended histories.
- Coverage bottleneck: 55.5% ± 3.3% to 19.0% ± 3.3%: ground-truth history-state regularization can sharply reduce success despite improving in-distribution accuracy.The negative result identifies insufficient coverage of corrective behaviors, rather than feature-learning architecture or objective, as the fundamental bottleneck.
4. A Simple Recipe for History-Conditioning in Imitation Learning
BPP addresses history-coverage mismatch by replacing full observation histories with a compact set of task-relevant keyframes detected by a VLM, while accounting for detection latency. This representation increases train-test overlap and reduces context-processing costs.
- Motivation and approach: BPP changes the history representation rather than the learning algorithm, replacing full histories with a small set of behaviorally salient events.The approach targets insufficient coverage between training and deployment histories without changing data collection or adding auxiliary regularization.
- Keyframe construction: Keyframes are defined by concise, task-specific criteria and detected with a VLM, such as button presses or opened drawers whose interiors are visible.The detector identifies semantically meaningful moments rather than uniformly retaining past observations.
- Keyframe construction: BPP retains event onsets as keyframes, avoiding duplicate detections when consecutive observations correspond to the same event.This rising-edge rule keeps the first frame of consecutive positive detections and separates distinct events with negative detections.
- Latency handling: Latency-masked keyframes simulate detector delays during training, so the policy conditions only on information realistically available at inference.The implementation uses a delay parameter Δ; VLM queries incur average latency of 3 to 5 seconds, with Δ=3 seconds during training.
- Efficiency: 41%: BPP reduces training time on Drawer Search compared with naïve history conditioning by processing fewer context frames.The reduction is attributed to using fewer context frames.
5. Experiments
Across real-world and simulation evaluations, BPP outperforms comparison methods by using compact, semantically meaningful history representations. Its gains include stronger progress tracking, task-dependent benefits over PTP, and improved data efficiency.
- Real-world results: BPP achieves nearly 70% higher average success than PTP in the real-world experiments.
- Behavioral analysis: Naïve history conditioning hurts performance in two of four real-world tasks and can overfit to spurious correlations from background, sensor, and actuator variation.
- Behavioral analysis: PTP significantly improves over naïve history on Mug Replacement and Stacking Puzzle but offers little benefit on Drawer Search and Marshmallows.
- Behavioral analysis: BPP systematically searches drawers without revisiting checked locations, while memoryless policies repeat actions or terminate prematurely.
- Simulation results: BPP significantly outperforms all non-oracle simulation comparisons and exceeds Oracle on Variable-Password.
- Ablation studies: With enough demonstrations, Naïve History matches BPP on Mug Replacement, but BPP reaches strong performance with far less data.
6. Discussion and Conclusion
BPP addresses insufficient history coverage by compressing observations into semantic keyframes, improving performance across simulation and real-world tasks. Its practical deployment remains constrained by external VLM latency and keyframe-detection errors.
- Discussion and Conclusion: BPP compresses observation histories into task-relevant keyframes, reducing distribution shift between training and deployment.The approach uses vision-language models to identify semantic events instead of conditioning on raw temporal sequences.
- Discussion and Conclusion: VLM inference latency constrains real-time deployment, especially for highly dynamic tasks.The limitation is highlighted for parts of Mug Replacement.
- Discussion and Conclusion: Systematic keyframe-detection errors can trigger premature state transitions, despite robustness to some errors in ablations.The paper gives misclassifying an empty grasp as an example of a false positive.
- Discussion and Conclusion: BPP suggests event-based robot learning as a route toward scaling semantic-milestone representations to multi-stage, long-horizon tasks.The stated future direction includes diverse, non-expert data.
A.1. Frozen Encoder Ablation
The frozen encoder ablation shows that historical observations require adapted visual representations rather than an encoder trained only on current observations.
- A.1. Frozen Encoder Ablation: Freezing the image encoder significantly reduces performance on Fixed Password Entering.The frozen encoder is initialized from a policy trained using current observations only.
- A.1. Frozen Encoder Ablation: The image encoder must extract different features from historical frames than from current observations.This conclusion follows from the performance drop under encoder freezing.
B.1. Policy Architecture
The policy architecture encodes four camera views and proprioception into tokens, then uses a transformer decoder to denoise actions. BPP adds only modest training-time overhead relative to current-observation training.
- B.1. Policy Architecture: Each timestep uses four camera views processed by a standard ResNet34 image encoder with camera-specific weights shared across timesteps.The resulting image features are flattened into tokens.
- B.1. Policy Architecture: Proprioception, the diffusion timestep, and the action to denoise are represented as additional tokens for the policy decoder.The architecture passes these tokens together with image-feature tokens.
- B.1. Policy Architecture: The transformer decoder denoises the action output.This role is stated in the architecture figure description.
- B.1. Policy Architecture: BPP adds only modest training-time overhead compared with current-observation-only training, whereas naïve history conditioning significantly increases training time.The comparison is reported for Drawer Search.
C. Task Specifications
The task suite spans real-world manipulation and simulated memory problems, with keyframes defined by task-specific milestone events. Evaluations vary in demonstrations, initial conditions, and success criteria.
- Task Specifications: Mug Replacement uses randomized mug placements, 900 demonstrations, and mug pickups as keyframes.The task requires swapping mugs between a coffee machine and plates.
- Task Specifications: The marshmallow task requires two successful handful transfers followed by a button press.A transfer counts when at least one marshmallow enters and remains in the red bowl.
- Task Specifications: Drawer Search uses six drawers, randomized search strategies, 200 demonstrations, and drawer-opening events as keyframes.Evaluations test six key positions three times each, for 18 evaluations.
- Task Specifications: Stacking Puzzle evaluates six piece orderings, uses piece pickups as keyframes, and reports completion score because total success is very low.BPP and PTP each achieve 2 out of 18 total successes, while other baselines achieve 0.
- Task Specifications: Simulation required substantially more data than real-world experiments to achieve reasonable behaviors.The paper attributes this likely to lower-quality data and greater behavior diversity.
- Task Specifications: Simulation includes hidden-state tasks such as inserting invisible lemons and entering passwords requiring variable amounts of memory.Lemon-release events define keyframes, while password tasks measure ordered button completion.
D. Real-World Task and Evaluation Details
The paper uses task-specific VLM prompts to detect semantic keyframes from robot observations. These prompts identify events such as grasping objects, dropping marshmallows, or opening drawers.
- Keyframe prompts: The Mug Replacement prompt detects when the gripper newly grasps a mug.It requires the gripper to completely grasp the mug in the current observation but not the previous one.
- Keyframe prompts: VLM prompts detect task-specific keyframes by comparing current and previous robot observations.The prompts ask for a binary YES/NO decision based on an event occurring between adjacent observations.
- Keyframe prompts: The Marshmallows prompt detects when marshmallows have just been dropped into the red bowl.The event is identified when the gripper is open over the bowl after being closed previously.
- Keyframe prompts: The Drawer Search prompt detects whether either wrist-mounted view shows an open drawer.Only the small white cabinets count as drawers for this decision.
- Keyframe prompts: The Stacking Puzzle prompt detects when the gripper newly picks up a piece.The gripper must completely grasp the piece in the current observation but not the preceding one.
E. Evaluation Filmstrips
Evaluation filmstrips visualize sequences of frames from real-world tasks, using a common height scale. The examples show successful and failed behaviors across drawer search, marshmallow, mug replacement, and stacking tasks.
- Filmstrip format: Filmstrips capture sequences of frames from evaluation episodes across the real-world tasks.All filmstrips use the same height, while each task’s longest sequence spans the page width.
- Task behaviors: Drawer Search filmstrips show BPP maintaining long-term context while baselines fail to progress through the drawer sequence.
- Task behaviors: Marshmallow filmstrips compare successful and failed executions.
- Task behaviors: Mug Replacement filmstrips compare BPP success with Naïve History failure.
- Task behaviors: Stacking Puzzle filmstrips show BPP successfully solving the long-horizon sorting task.