Instruction-driven history-aware policies for robotic manipulations

• The motivating gaps include hidden object states and action history in sequential tasks, precise control under occlusion, and fine-grained vision-action-text alignment across task changes.Existing methods often rely on current observations or compress instructions into global vectors that can lose object-level information.Source: §1 · p. 1
• Hiveformer jointly models language, past and current visual observations, proprioception, and multi-camera spatial relations before predicting 7-DoF actions.A multimodal transformer processes the tokens, with a UNet decoder predicting position, rotation, and gripper state.Source: §1 · pp. 1–2
• Hiveformer significantly outperforms state-of-the-art models in single-task, multi-task, and multi-variation RLBench setups, and one model covers 74 tasks.The evaluation also groups the tasks into nine categories.Source: §1 · p. 2
• Hiveformer generalizes to new task variations and human-written instructions, while simulator pretraining boosts performance when real-robot demonstrations are scarce.The model is evaluated on a real robot after finetuning.Source: §1 · p. 2
• Ablations indicate that history helps long-term tasks, while multi-view input is most helpful for precision-demanding tasks and visual occlusions.Source: §1 · p. 2

Vision-based robotic manipulation

• Learning-based manipulation grew from single-task networks [29] to multi-task methods [32] [33], using reinforcement learning or behavioral cloning from demonstrations.Demonstrations may come from humans, robots, or play interactions; this work uses behavioral cloning on scripted RLBench demonstrations [19].Source: §2 · pp. 2–3

Instruction-driven vision-based robotic manipulation

• Most instruction-driven manipulation works [6] [8] [9] [10] are stateless, whereas this work jointly models language instructions, history, and multi-view observations.Prior approaches include grounding referring expressions and spatial relations, hierarchical sub-goal decomposition, instruction data, pre-trained vision-language models, and multi-camera fusion with known camera parameters.Source: §2 · p. 3

Transformers

• Transformers support multimodal action prediction using prior observations in navigation [74] [75] [76], and have also been used for supervised reinforcement learning and generalist agents [72] [77].This work explores transformers for instruction-driven, history-aware robotic manipulation.Source: §2 · p. 3

Problem Definition

• Macro steps mark gripper-state changes or near-zero joint velocities; inverse-kinematics trajectories between them reduce episodes from hundreds of small steps to typically fewer than 10.A motion planner executes actions in RLBench.Source: §3 · p. 3
• At each step, observations comprise RGB images and corresponding point clouds aligned pixel-by-pixel, with 3D points projected into world coordinates using camera depth and known intrinsics and extrinsics.Source: §3 · p. 3
• The setup uses three 128 × 128 camera views from the wrist, left shoulder, and right shoulder.Source: §3 · p. 3
• Actions comprise the gripper’s Cartesian position, quaternion rotation relative to the base frame, and a boolean open-or-closed state; an object is grasped between the fingers when the gripper closes.Source: §3 · p. 3

4 Our Model: Hiveformer

• Hiveformer combines feature encoding, a multimodal transformer, and action prediction to predict the next rotation, gripper state, and position.A CNN predicts rotation and gripper state, while a UNet decoder predicts position.Source: §4 · p. 4

4.1 Feature Encoding

• A frozen CLIP language encoder provides instruction-token embeddings, while per-camera RGB, point-cloud, and proprioception inputs are encoded with gripper-position attention maps as visual features.The visual encoder uses a UNet, and camera-id and patch-location embeddings distinguish visual tokens.Source: §4.1 · p. 4

4.2 Multimodal transformer

• The multimodal transformer uses distinct attention layers to relate current visual tokens to instruction and history context, and to relate patches across camera views.Cross-attention models inter-modal relationships between current visual tokens and instruction-conditioned context; self-attention models relationships among multi-camera patch tokens.Source: §4.2 · p. 5

4.3 Action Prediction

• The model concatenates transformer outputs with encoded visual representations to form a feature map for predicting the next action.Position is handled separately from rotation and gripper state, using point clouds.Source: §4.3 · p. 5
• Rotation and gripper state are regressed from CNN-decoded features, while position is decomposed into an expected point on the point cloud and an offset.The offset allows prediction of virtual points outside the point cloud’s convex hull.Source: §4.3 · p. 5
• A learnable embedding represents task and step, and the task identity is predicted from the instruction.The embedding is defined over tasks and maximum episode length.Source: §4.3 · p. 5

4.4 Training and Inference

• Behavioral cloning on successful RLBench demonstrations combines gripper-action MSE and task-classification cross-entropy; current-observation patches are randomly zeroed with probability 0.1.Masking encourages reliance on observation history when the current view alone contains sufficient information.Source: §4.4 · pp. 5–6

5.1 Experimental Setup

• The study compares models on 10 RLBench tasks using 100 demonstrations per task unless noted, and also evaluates 74 tasks using success rates over 500 unseen episodes.The 74 tasks are grouped into nine categories; the study considers single-task, multi-task, and multi-variation settings.Source: §5.1 · p. 6, §5 · p. 6

5.2 Ablations

• Ablations show gains from adding point clouds and gripper position, history and observation masking, spatial patch tokens, and instruction-conditioned cross-attention.The instruction-conditioned UNet baseline achieves 73.2%; reported gains are 3% from point clouds and gripper position, 4.5% from history, 0.5% from masking, 2.2% from patch tokens, and 3.8% from cross-attention.Source: §5.2 · pp. 6–7

5.3 Comparison with State of the Art

• On 10 RLBench tasks, Hiveformer consistently improves over ARM [12] and Auto-λ [15] in single-task evaluation and outperforms Auto-λ by 14% in multi-task evaluation.In multi-task evaluation, Hiveformer treats tasks equally, whereas Auto-λ dynamically adjusts their weights.Source: Table 2 · p. 7, §5.3 · p. 7
• Across 74 RLBench tasks, full Hiveformer outperforms Auto-λ [15] across task groups; history helps on Long-term, Tools, and Planning, while multiple views help on fine-control and occlusion tasks.Performance is relatively poor on Long-term tasks exceeding 10 steps, which are more exposed to distribution shift and accumulated errors.Source: Table 3 · p. 7, §5.3 · p. 7
• With 10 demonstrations per variation, Hiveformer reaches 71.1% success on unseen Push Buttons variations and 49.8% on unseen Tower variations.Instructions are necessary to generalize to unseen variations; the model also performs well on human-written instructions despite training on synthetic instructions.Source: Table 4 · p. 8, §5.3 · p. 8

5.4 Real-robot Experiments

• Simulator pretraining followed by real-robot finetuning significantly improves push-button performance, especially on unseen variations, while training from scratch is prone to overfitting on seen variations.The experiments used a UR5 arm, 10 task variations, and 10 real-robot demonstrations per variation.Source: §5.4 · p. 8

6 Conclusion

• Hiveformer outperformed the state of the art across RLBench single-task, multi-task, and multi-variation generalization settings.The conclusion reports effectiveness across all three evaluation settings but provides no quantitative comparison here.Source: §6 · p. 8
• Hiveformer jointly models natural-language instructions, multi-camera views, and observation-action history for robotic manipulation.The conclusion identifies this combination as the model's core design.Source: §6 · p. 8
• On a real robot, Hiveformer generalized to unseen variations and human-written instructions, though it performed worse on human-written instructions.The model was trained only on synthetic instructions.Source: §6 · p. 8
• The model's limitations include transformer computational cost that grows quadratically with input sequence length and potential exposure bias from behavioral-cloning training.The conclusion identifies both as limitations.Source: §6 · p. 8
• Future work could improve long-term-task efficiency with hierarchical models, incorporate reinforcement learning, and train on automatically generated human-written instructions.The conclusion presents these as possible directions for addressing efficiency, training, and human-instruction performance.Source: §6 · p. 8

Appendix

• The appendix covers architecture, RLBench task categorization, experimental details, additional ablations, and quantitative and qualitative real-robot experiments.Table 5 reports success rates for the push buttons task on real robots.Source: Table 5 · p. 8, Appendix · p. 14

A Model Architecture

• The image encoder has six convolutional layers and maps an H × W × 3 image to an H/16 × W/16 × 16 feature map.The first two layers use stride 1 and 8 and 16 output channels; the remaining four use stride 2 and 16 channels, with group normalization and LeakyReLU.Source: A Model Architecture · p. 14
• The U-Net decoder stacks four convolution-and-upsampling blocks with encoder residual connections to recover image size and generate a point-cloud heatmap.Each convolution uses 16 output channels, a 3 × 3 kernel, and stride 1; each upsampling layer uses factor 2 and bilinear sampling.Source: A Model Architecture · p. 14
• A separate two-layer CNN feeds an average-pooled, flattened feature vector to a feedforward network that predicts a 4D quaternion and 1D gripper state.Both convolutional layers use 3 × 3 kernels, stride 2, 64 output channels, group normalization, and LeakyReLU.Source: A Model Architecture · p. 14

B Categorization of RLBench Tasks

• Of RLBench2’s 106 supported tasks, 74 were evaluated and manually grouped into nine categories because demonstration generation failed for 32 tasks.The failures were due to issues with task scripts and the motion planner.Source: B Categorization of RLBench Tasks · p. 14
• Planning tasks involve multiple sub-goals, Tools tasks require grasping objects to interact with targets, and Long term tasks take more than 10 macro-steps.Source: B Categorization of RLBench Tasks · p. 14
• Rotation-invariant tasks need no gripper-rotation changes, Motion planner tasks require precise grasping, and Multimodal tasks permit multiple trajectories across large target affordance areas.Motion-planner failures are noted as common for precise-grasping tasks [81].Source: B Categorization of RLBench Tasks · p. 14
• Precision tasks involve precise object manipulation, Screw tasks require screwing an object, and Visual Occlusion tasks involve large objects that occlude some views.Source: B Categorization of RLBench Tasks · pp. 14–15

C Experimental Details

• For multi-variation experiments, push-buttons and tower tasks each use 100 training variations and 100 different test variations.These tasks were selected because their variations are easy to construct.Source: C Experimental Details · p. 15
• The modified RLBench motion planner retries with different seeds up to 10 times to find a trajectory when its default planning fails.This reduces the impact of planner failures when successful trajectories exist in 3D space.Source: C Experimental Details · p. 15
• The push-buttons task varies button colors and press order, while the tower task varies colored-block stacks and their instructed order.Synthetic instructions are generated for each variation using RLBench sentence templates for push buttons and generated instructions for tower.Source: C Experimental Details · p. 15
• Testing includes 162 human-written instructions from 8 native English speakers, assessed over 10 randomized-object episodes per instruction.The collection contains 63 instructions for 51 push-buttons variations and 99 instructions for 99 tower variations.Source: C Experimental Details · p. 15
• Human-written test instructions include unseen verbs, formulations, longer sentences, and color references, probing variation beyond synthetic wording.Examples include alternative color descriptions such as “darker blue” and “lighter blue.”Source: C Experimental Details · p. 15

D.1 Comparison with Additional State-of-the-Art Approach

• 88.3% vs. 63.9% (single-task) and 83.8% vs. 47.8% (multi-task) average success rates favor the proposed approach over LanCon-Learn’s original method.With history, LanCon-Learn reaches 72.9% in single-task and 57.4% in multi-task settings. The comparison covers 10 tasks.Source: Table 6 · p. 16, §D.1 · p. 16

D Experiments on the simulator RLBench

• Further simulator ablations examine instruction encoding across seen or unseen variations and synthetic, corrupted, or real instructions.The reported comparison includes LanCon-Learn and an instruction-encoding ablation in the multi-variation setting.Source: Table 7 · p. 16, D Experiments on the simulator RLBench · p. 16

D.1 Comparison with Additional State-of-the-Art Approach

• LanCon-Learn uses ground-truth object poses, GloVe instruction embeddings, and a bi-directional LSTM; the authors add RGB object references and a previous-step gripper state.The previous-step gripper state supplies a history mechanism for the baseline comparison.Source: §D.1 · p. 16

D.2 Additional Ablations on Multi-variation Setting

• 1.7% (LanCon-Learn) vs. 86.3% (our approach) on unseen “push buttons” variations with synthetic instructions; replacing CLIP with BERT gives 40.2%.The passage attributes LanCon-Learn’s larger gap to GloVe embeddings’ difficulty differentiating colors.Source: §D.2 · p. 16
• Without history, the model struggles to track which buttons were pressed and infer the next target from the current observation, hurting long-term “push buttons” performance.The passage says the importance of history is confirmed by comparisons with and without history for LanCon-Learn and the proposed model.Source: §D.2 · p. 16
• One-hot variation-index instructions improve seen-variation tower performance by 19.4% but prevent generalization to unseen variations.For corrupted instructions on unseen variations, created by replacing color references with unseen synonyms, CLIP language-encoder baselines have a much smaller performance drop than other encoders.Source: §D.2 · p. 17
• Global language embeddings significantly reduce performance; averaging token embeddings yields 0.97 average cosine similarity for instructions with the same actions in different orders.The passage links this similarity to averaged embeddings’ poor representation of action order.Source: §D.2 · p. 17

E Experiments on Real-robot

• The robot pressed buttons in the instructed order across unseen variations, remaining robust to changes in button appearance, shape, occlusion, and scene dynamics.Challenges included a flipped black button resembling a white one, two white buttons, a button moved mid-task, gripper occlusion, and a changed table appearance.Source: E Experiments on Real-robot · pp. 17–18, Figure 6 · p. 19
• The real-robot setup used two Intel RealSense RGB-D cameras and button-sequence demonstrations, and the model was finetuned for 8k iterations.Images were cropped and downsampled to 128×128; objects were randomly placed and actions executed at 10 Hz.Source: E Experiments on Real-robot · p. 17
• Using observation-action history, the robot avoided the flipped black button and selected the correct white button, and coped with two white buttons that could yield a mean-position prediction.The policy also remained robust when the white button was moved after the yellow button was pressed, a perturbation absent from training sequences.Source: E Experiments on Real-robot · pp. 17–18