Source-linked AI summary
Instruction-driven history-aware policies for robotic manipulations
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia, Makarand Tapaswi, Ivan Laptev, Cordelia Schmid
TL;DR
Robotic manipulation requires precise control, memory of past interactions, and generalization from language instructions across tasks and environments. Hiveformer addresses these needs with a multimodal transformer combining instructions, multi-view observations, and history. It outperforms state-of-the-art methods across three RLBench settings, scales to 74 tasks, and generalizes to unseen variations and human-written instructions on a real robot.
Problem
Instruction-following manipulation remains challenging because robots need fine-grained control, long-term memory, and generalization to unseen tasks and environments.
Method
Hiveformer jointly models language instructions, multi-view visual observations, proprioception, and observation-action history with a multimodal transformer for action prediction.
Results
Hiveformer outperforms state-of-the-art models in single-task, multi-task, and multi-variation RLBench settings, covering 74 tasks and generalizing to unseen human-written instructions.
Takeaways & Limitations
A single Hiveformer trained with synthetic instructions can solve multiple tasks and variations, generalize to unseen human-written instructions, and perform well on a real robot after finetuning.
Abstract
from arXiv · showhide
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.
1 Introduction
Hiveformer targets instruction-following manipulation that must remember history, integrate multiple views, and generalize across tasks and variations. It combines these inputs in a multimodal transformer and outperforms prior methods across RLBench settings, including 74 tasks and unseen instructions.
- Current-observation methods struggle with sequential tasks requiring hidden object-state tracking or memory of previously executed actions.
- Single-view manipulation is difficult under occlusion and precise end-effector placement, while global instruction vectors can lose fine-grained object information.
- Hiveformer combines language tokens, past and current visual observations, proprioception, and a multimodal transformer to predict 7-DoF actions.
- 74 RLBench tasks are evaluated across single-task, multi-task, and multi-variation settings, with results analyzed in 9 task categories.
- Hiveformer outperforms state-of-the-art models, generalizes to unseen human-written instructions, and performs well on a real robot after simulator pretraining and finetuning.
2 Related Work
Prior work moved from manually designed manipulation toward learning-based single-task and multi-task systems. These methods use shared representations, meta-learning, goal conditioning, inverse reinforcement learning, reinforcement learning, or behavioral cloning, while task and variation definitions distinguish shared skills from changing objects, attributes, or order.
- Learning-based manipulation methods address increasing variation in objects and environments beyond earlier manually designed visual-servoing systems.
- In RLBench, a task can contain variations sharing skills while differing in objects, attributes, or action order.
- Multi-task approaches include shared features, meta-learning, goal-conditioned learning, and inverse reinforcement learning.
- Training methods generally fall into reinforcement learning from rewards or behavioral cloning from demonstrations.
3 Problem Definition
The problem is instruction-conditioned robotic control from language, multi-camera observations, point clouds, and action history. RLBench represents actions as gripper pose and state, while macro steps reduce long trajectories to key control points.
- The model receives a natural-language instruction represented as word-level inputs together with visual and action information.
- Each observation contains RGB images and point clouds aligned across multiple cameras.
- Macro steps mark gripper-state changes or near-zero joint velocities, reducing episodes from hundreds of small steps to typically fewer than 10.
- RLBench actions consist of Cartesian gripper position, quaternion rotation, and a binary open-or-closed gripper state.
- The setup uses multiple camera views to support instruction-driven manipulation under the defined observation and action representation.
4 Our Model: Hiveformer
Hiveformer encodes instructions, multi-camera observations, proprioception, and history as tokens, then uses specialized attention to model cross-modal and cross-view relationships. Its decoders convert the contextualized representation into the next position, rotation, and gripper state.
- Architecture: Hiveformer has feature-encoding, multimodal-transformer, and action-prediction modules.
- Action Prediction: The action head predicts the next gripper rotation and state with a CNN decoder and predicts position through a separate point-cloud-based module.
- Feature Encoding: A pretrained language encoder converts instructions into tokens, while RGB images, point clouds, proprioception, camera identity, patch location, and step identity supply visual tokens.
- Feature Encoding: The visual representation includes a binary attention map marking the gripper center and combines image, point-cloud, and proprioceptive inputs.
- Multimodal Transformer: The transformer uses cross-attention for instruction–vision relationships and self-attention for relationships among patches from multiple camera views.
- Multimodal Transformer: History tokens are incorporated with current observations so the transformer can model dependencies between current observations, prior context, views, and instructions.
W2 GeLU
The model predicts robotic actions by combining transformer features with point-cloud-based position estimation and separate rotation/gripper-state decoding. It is trained with behavioral cloning, task classification, and current-observation masking to encourage use of history.
- Action prediction: The action module concatenates transformer and visual features before predicting the next position, rotation, and gripper state.Position prediction uses a separate point-cloud module for accurate fine-grained positioning.
- Action prediction: Rotation and gripper state are decoded with a CNN, while position is decomposed into an expected point and an offset.The offset supports virtual target points outside the point cloud’s convex hull.
- Action prediction: Position is computed as an expected location over camera-specific point clouds using predicted attention probabilities.Each attention value represents the probability of reaching its corresponding point.
- Task conditioning: The model predicts task identity from the instruction and current step id using a softmax over task classes.
- Training: Behavioral cloning trains the model on successful demonstrations with a loss combining gripper-action mean-square error and task-classification cross-entropy.
- Training: Randomly masking 10% of current-observation features encourages action prediction to use past observations rather than only the current view.Masked patches must be completed from history when the unmasked observation is insufficient.
5 Experiments
Experiments evaluate Hiveformer across RLBench single-task, multi-task, and multi-variation settings, with ablations isolating history, multi-view processing, and architectural choices. The model is compared across 10 and 74 tasks, unseen variations, and real-robot push-button experiments.
- Experimental settings: The evaluation covers single-task, multi-task, and multi-variation settings, including tests on unseen task variations.Single-task models specialize per task, multi-task models share one model across tasks, and multi-variation models generalize to new variations.
- Ablations: 73.2% success rate is achieved by the instruction-conditioned current-observation baseline before adding the multimodal transformer.This baseline uses a UNet architecture and current visual observations without history.
- Ablations: 3% total improvement follows adding point clouds and gripper position to the feature encoding.These additions are evaluated in the R3 and R4 variants.
- Ablations: 4.5% absolute gains come from history, masking observations adds 0.5%, and patch-tokenization adds another 2.2%.The ablations study prior observations, training-time masking, and patch tokens that encode fine-grained spatial information.
- Ablations: 3.8% further improvement results from replacing self-attention with cross-attention to condition on instruction and history context.The replacement is described as the final attention-design ablation.
- Benchmark comparisons: Hiveformer consistently outperforms Auto-λ across 74 RLBench tasks and performs especially well on fine-control and visual-occlusion categories.History helps Long-term, Tools, and Planning groups, while multiple views help Screw, Precision, and Visual Occlusion categories; performance is relatively poor on Long-term tasks exceeding 10 steps.
- Benchmark comparisons: A single model trained across tasks performs only slightly worse than separate task-specific models, while instructions are important in the multi-task setting.The multi-task comparison supports sharing one model across multiple tasks.
- Real-robot evaluation: Pretraining on RLBench significantly improves real-robot push-button performance, especially on unseen variations, whereas training from scratch tends to overfit seen variations.The real-robot setup uses a 6-DoF UR5, two cameras, 10 task variations, and 10 demonstrations per variation.
6 Conclusion
The paper concludes that Hiveformer jointly models instructions, multi-camera views, and history for instruction-driven manipulation, with evaluations spanning RLBench and real-robot deployment. It also identifies computational cost, exposure bias, and weaker performance on human-written instructions as limitations.
- Table 5 reports success rates for the push buttons task on real robots.
- Hiveformer jointly models instructions, multiple camera views, and history for instruction-driven robotics manipulation.
- The model is evaluated across single-task, multi-task, and multi-variation RLBench settings and deployed on a real robot.
- Transformer computation grows quadratically with input sequence length, while behavioral cloning may suffer from exposure bias.
- The model is trained only on synthetic instructions and performs worse on human-written instructions.
A Model Architecture
The architecture uses a convolutional U-Net encoder to convert images into compact feature maps and a U-Net decoder to predict position heatmaps on point clouds.
- The image encoder uses six convolutional layers with 3x3 kernels and produces a feature map reduced by a factor of 16 in height and width.The first two layers use stride 1; the remaining four use stride 2, with normalization and LeakyReLU activations.
- The encoder outputs channels of sizes 8 and 16 in its first two layers, followed by four 16-channel layers.
- The decoder combines convolutional outputs with residual connections from corresponding encoder layers.
- Four convolutional and bilinear upsampling blocks recover spatial resolution and generate a point-cloud position heatmap.
B Categorization of RLBench Tasks
The study groups 74 RLBench tasks into nine categories based on their main challenges, including planning, tools, screwing, and visual occlusion. Figure 3 highlights the Push Buttons and Tower tasks used for multi-variation experiments.
- 74 RLBench tasks are manually grouped into 9 categories according to their key challenges.
- The Planning group contains tasks with multiple sub-goals, while the Tools group requires grasping an object to interact with a target.
- Figure 3 shows Push Buttons on the left and Tower on the right as the multi-variation tasks.
- The Screw group requires screwing an object, and the Visual Occlusion group contains tasks where large objects block some views.
C Experimental Details
The experiments cover task and variation generalization in RLBench, with Push Buttons and Tower providing instruction-defined variations. Additional comparisons evaluate LanCon-Learn and instruction-encoding ablations.
- The motion planner is run with different seeds up to 10 times to reduce failures caused by unsuccessful target-pose planning.
- Push Buttons varies button colors and press order, while Tower varies the ordered stacking of colored cubes.
- The appendix includes comparisons with LanCon-Learn and ablations of instruction encoding for seen and unseen variations with synthetic, corrupted, or real instructions.
- LanCon-Learn reaches 63.9% without history, 72.9% with history, and 88.3% for Hiveformer in single-task experiments.
- LanCon-Learn reaches 47.8% without history, 57.4% with history, and 83.8% for Hiveformer in multi-task experiments.
D.2 Additional Ablations on Multi-variation Setting
The multi-variation ablations show that history and language representations are central to solving long-term, instruction-conditioned tasks, especially when color and action order must be distinguished.
- 86.3% versus 1.7%: Hiveformer substantially outperforms LanCon-Learn on unseen-variation push-buttons with synthetic instructions.Replacing CLIP with BERT in Hiveformer lowers performance to 40.2%.
- Removing history makes pushing-buttons difficult because the model cannot infer which buttons were already pressed or which is the next target.
- Averaging word-token embeddings weakens action-order representation, with instructions for the same actions in different orders reaching an average cosine similarity of 0.97.
E Experiments on Real-robot
The real-robot experiments use two fixed RGB-D cameras with a UR5 arm and test instruction following under visual ambiguity and scene perturbations. History-aware predictions help the robot select the intended button and remain successful after objects move or appear ambiguous.
- History lets the robot press the correct white button after the gripper accidentally flips another button, leaving two buttons that look white.
- The robot handles two white buttons without predicting a mean position between them, addressing the ambiguity of multimodal target locations.
- The learned policy remains robust when a ruler moves the white button after the yellow button is pressed, despite such perturbations being unseen during training.