PaperlayerView exact revision on arXiv

Source-linked AI summary

VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors

Yifeng Zhu, Abhishek Joshi, Peter Stone, Yuke Zhu

arXiv:2210.11339v2cs.RO

TL;DR

Vision-based neural imitation-learning methods for manipulation can be susceptible to distribution shifts and observation noise. VIOLA builds object-centric representations from pretrained object proposals and uses a transformer policy; it outperforms baselines in simulation and the real world, including BC-RNN by 46.7% success rate on average.

  • Problem

    Neural-network approaches to vision-based manipulation are susceptible to distribution shifts and observation noise.

    Source: §2 · p. 2

  • Method

    VIOLA represents image regions using pretrained object proposals and uses a transformer policy to reason over objects and their relations for action prediction.

    Source: §3.1 · p. 3, §3.3 · p. 5

  • Results

    46.7% success rate on average: VIOLA outperforms BC-RNN, and it outperforms state-of-the-art baselines in simulation and the real world.

    Source: §4.2 · p. 8, §5 · p. 8

  • Takeaways & Limitations

    Ablative studies validate that VIOLA’s model components impact policy performance.

    Source: §5 · p. 8

    VIOLA’s behavioral-cloning setup assumes access to a set of demonstrations.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA outperforms the state-of-the-art imitation learning methods by $45.8\%$ in success rate. It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making. More videos and model details can be found in supplementary material and the project website: https://ut-austin-rpl.github.io/VIOLA .

1 Introduction

VIOLA builds object-centric representations from general object proposals and uses a transformer policy to select task-relevant visual cues for action prediction. The paper reports stronger simulation performance and successful real-robot manipulation, including coffee-making.

1 Introduction

  • Object-centric priors are intended to help policies discard spurious visual dependencies and generalize, addressing the brittleness of end-to-end imitation learning under covariate shifts and perturbations [5] [6] [7] [8] [9] [10] [11].The approach treats objects operationally as disentangled visual concepts that inform robot decisions.Source: §1 · p. 1
  • Prior object-aware policies were limited to simple control [6], single-object manipulation [15], or costly detection annotations [16]; VIOLA uses general object proposals as priors [18].The proposals are localized image bounding boxes that capture general objectness across appearance variations and have supported downstream vision tasks [19] [20] [21].Source: §1 · pp. 1–2
  • VIOLA uses a pre-trained Region Proposal Network to extract proposal features, then a Transformer encoder uses multi-head self-attention to focus on task-relevant regions for action generation.The proposal features become factorized object-centric representations and are converted into discrete tokens for supervised imitation learning.Source: §1 · p. 2, Figure 1 · p. 2
  • 45.8%: VIOLA outperforms the most competitive baseline in success rate and achieves 60% success on multi-stage coffee-making, while baseline methods fail entirely.Simulation evaluations cover training-distribution performance and variations in object placements, distracting objects, and camera pose; the three real-world tasks use 50 demonstrations.Source: §1 · p. 2

2 Related Work

Prior visuomotor imitation methods face limitations in handling high-dimensional observations, distribution shifts, and unfamiliar objects. VIOLA builds on object-centric imitation learning by using pre-trained region proposals as priors for manipulation policies.

Imitation Learning (IL) for Manipulation

  • Non-parametric methods such as DMP and PrMP learn behaviors from few demonstrations but are typically open-loop, while neural policies can be vulnerable to distribution shifts and observation noise [1] [2] [3] [4] [27].Prior object-centric imitation learning addressed these issues but focused on single object instances or required costly annotations for detector pre-training.Source: §2 · p. 2
  • VIOLA uses a pre-trained RPN to provide object proposals as priors for end-to-end imitation policies, targeting visual robustness and tasks with complex multi-object interactions.This follows the object-centric imitation learning idea while addressing prior approaches’ limitations around single instances or costly detection annotations.Source: §2 · p. 2

Visual Representations for Visuomotor Policies

  • Bounding-box representations require fine-tuned or category-specific detectors and generalize poorly to unknown objects, while raw-observation policies can suffer covariate shift and causal confusion [2] [3] [6] [16] [28] [29].Spatial attention [30] [31] [32] and affordances [33] [34] [35] [36] [37] [38] add inductive biases, but are designed for specific motion primitives and limit behavioral diversity.Source: §2 · p. 3

Object-Centric Representation

  • Robotics object abstractions such as poses and bounding boxes rely on prior object knowledge and may miss fine-grained details, while unsupervised discovery struggles with complex scenes [28] [16] [29] [39] [40] [41] [42].Motivated by RPN effectiveness on out-of-distribution images [21], VIOLA uses object proposals to scaffold object-centric representations for manipulation.Source: §2 · p. 3

3 Approach

VIOLA formulates vision-based manipulation as behavioral cloning over object-centric representations built from visual proposals and contextual signals. A transformer policy uses these representations to predict closed-loop actions.

3.1 Problem Formulation

  • The task is an MDP over raw sensory observations and motor commands; behavioral cloning learns a policy from teleoperated demonstration trajectories to maximize discounted return.Source: §3.1 · p. 3

3.2 Object-Centric Representation

  • Without ground-truth object states, VIOLA treats top K pre-trained RPN proposals as approximate objects and combines region appearance and position with workspace, eye-in-hand, and proprioceptive context.Visual features come from a learned ResNet-18 spatial feature map with ROI Align; proposal-box coordinates are sinusoidally encoded.Source: §3.1 · p. 3, §3.2 · pp. 4–5
  • Temporal composition of H+1 per-step features with temporal positional encodings captures object-state changes and increases proposal recall; ablations indicate ordering matters, especially with a transformer policy.Source: §3.2 · p. 5

3.3 Transformer-based Policy

  • The transformer treats region and context features as tokens, uses a learnable action token to attend to task-relevant regions, and maps its output through an MLP and GMM action head.VIOLA uses K = 20 proposals in simulation, K = 15 in the real world, and H = 9; training includes color augmentation, pixel shifting, and random erasing.Source: §3.3 · pp. 5–6

4 Experiments

The experiments compare VIOLA with imitation-learning baselines in simulation and on real robots, test object-centric design choices, and assess practical deployment.

4.1 Experimental Setup

  • The tests span object-placement variation, long-horizon execution, distractors, and camera jitter while preserving task semantics.Simulation includes Sorting, Stacking, and BUDS-Kitchen; real-robot tasks include Dining-PlateFork, Dining-Bowl, and Make-Coffee. Canonical tests match demonstration distributions, while Placement, Distractor, and Camera-Jitter vary specific conditions.Source: §4.1 · p. 6

4.2 Experiment Results.

  • 46.7%: VIOLA outperforms BC-RNN by this average success-rate margin across the real-world tasks.In Make-Coffee, VIOLA robustly grasps k-cups or opens the coffee machine, while BC-RNN can miss grasps or mistakenly release the gripper.Source: Figure 4 · p. 7, §4.2 · p. 8
  • 50.8% in Canonical and 44.1% across three testing variants: VIOLA’s success-rate advantages over BC-RNN.The authors also report that OREO does not consistently outperform simple BC, while temporal modeling benefits the compared methods in complex vision-based manipulation.Source: §4.2 · p. 7
  • VIOLA outperforms VIOLA-Patch especially on long-horizon BUDS-Kitchen, suggesting complete object coverage from proposal regions is important.VIOLA-Patch is comparable in some setups, showing that a transformer can attend to task-relevant patch regions even when patches are not object-based.Source: §4.2 · p. 7
  • Temporal windows make transformer performance comparable to BC-RNN, positional encoding improves it beyond top baselines, and Random Erasing yields the highest success across evaluation setups.Visual features alone improve robustness to Distractor; adding visual and positional features helps Canonical and Distractor but hurts Placement and Camera-Jitter, where the authors hypothesize overfitting to proposal-box locations.Source: §4.2 · p. 7
  • Attention visualizations show VIOLA attends to the k-cup when grasping and to the k-cups, gripper, and coffee machine when closing the machine.The visualized top-attended regions support the policy’s spatial reasoning across observations.Source: Figure 5 · p. 8, §4.2 · p. 8

5 Conclusion

VIOLA builds object-centric representations from general object proposals and uses a transformer policy to generate actions, with superior performance to state-of-the-art baselines in simulation and the real world. The conclusion also identifies limitations in offline learning, proposal-model adaptation, and monocular RGB inputs, with online learning and depth as future directions.

5 Conclusion

  • VIOLA outperforms state-of-the-art baselines in simulation and the real world, and ablation studies validate the impact of its model components.The conclusion does not provide numerical results for these comparisons.Source: §5 · p. 8
  • General object proposals encode region-level visual and positional features alongside global scene and robot-state context, while a transformer policy selects task-relevant regions for action generation.Source: §5 · p. 8
  • VIOLA inherits offline-demonstration limitations, and its unadapted pre-trained RPN may fail under distribution shifts from natural images; online learning and RPN fine-tuning are proposed directions.Source: §5 · p. 8
  • Monocular RGB lacks object depth and can leave background elements such as tables entangled with object representations, limiting generalization to visual changes; depth images are proposed to address this.The conclusion specifically gives table-texture changes as an example of a challenging visual change.Source: §5 · p. 8

A Additional Implementation Details

This appendix provides model implementation details beyond those described in the main text.

A Additional Implementation Details

A.1 Model Details

The implementation specifies shared sensor inputs and their visual and proprioceptive encoders, a transformer policy with positional encodings, and the augmentation and training procedures.

Region and Context Features Computation

  • Workspace features use ResNet-18 and ROI Align on a 16 × 16 spatial map to produce 6 × 6 pooled features, while context, eye-in-hand, and proprioceptive features use Spatial Softmax or linear projection.The pooled workspace features are linearly projected to match the input-token size; eye-in-hand images use ResNet-18 followed by Spatial Softmax.Source: §A.1 · p. 13

Sensor Modalities

  • All models use workspace and eye-in-hand RGB images, joint configuration, and binary gripper state; prior work reports that the eye-in-hand view contributes significantly to visuomotor policy performance [27] [60] [61].Source: §A.1 · p. 13

Neural Network Details

  • The policy uses a 4-layer Transformer with 6 attention heads, 1024-unit fully connected layers, and a 5-mode GMM head, matching the mode count in Mandlekar et al. [27].Source: §A.1 · p. 13

Positional Encoding

  • Region positional features use bounding-box coordinates, and temporal encodings have token dimensionality D and frequency 10, chosen for the shorter input sequence.The positional encodings use sine and cosine functions across dimensions.Source: §A.1 · p. 13

Data Augmentation

  • Color jittering and pixel shifting are applied to VIOLA and baselines, while random erasing is limited to VIOLA and its variants because it lowered BC-RNN performance.Color jittering is applied to 90% of trajectories, with 10% unchanged; the procedure also uses four-pixel shifts and small Gaussian noise on proposal positions.Source: §A.1 · pp. 13–14

Training Details

  • VIOLA trains for 50 epochs with batch size 16, learning rate 10^-4, negative log likelihood, AdamW, cosine annealing, and gradient clipping; checkpoints use lowest demonstration-data loss.Gradient clipping is 10 except for BUDS-Kitchen and Make-Coffee, which use 0.1; baselines train for 500 epochs (BC) or 600 epochs (BC-RNN) and use the same checkpoint criterion.Source: §A.1 · p. 14

B Additional Experiments

This section presents two additional experiments intended to support aspects of the paper’s design.

B Additional Experiments

B.1 Choice of K in simulation

The simulation experiments use K=20 object proposals: recall exceeds 70% at IoU=0.5, while each additional five proposals beyond 20 increases recall by only 1%.

B.1 Choice of K in simulation

  • Recall exceeds 70% at K=20 and IoU=0.5, while each additional five proposals beyond 20 raises recall by only 1%, motivating the simulation choice of K=20.Recall measures coverage of scene objects, including robots, by RPN proposals.Source: Table 3 · p. 14, §B.1 · p. 14

B.2 Visual Feature Design

VIOLA learns spatial visual features from scratch to produce actionable representations for continuous control. Ablations show that pre-trained FPN features do not outperform this approach, even with fine-tuning.

B.2 Visual Feature Design

  • VIOLA learns a spatial feature map from scratch to extract actionable visual features that are informative for continuous control.Source: §B.2 · p. 14
  • The convolutional encoder for spatial feature maps has 14 MB of trainable parameters, compared with about 200 MB for the FPN.Source: §B.2 · p. 15
  • Pre-trained FPN features without fine-tuning perform worse overall, while fine-tuning nearly matches but does not surpass learning spatial features from scratch.The comparison is reported in Table 4.Source: Table 4 · p. 15, §B.2 · p. 15

C Environment Details

The environment details specify a shared robot-and-camera setup, task goals, demonstration collection, success criteria, and evaluation horizons across simulation and real-world tasks.

Overall setup

  • All tasks use a 7-DoF Franka Emika Panda arm, an Operational Space Controller, a binary parallel-gripper command at 20Hz, and Kinect Azure and Intel Realsense D435i cameras.Kinect Azure is the workspace camera and Intel Realsense D435i is the eye-in-hand camera.Source: C Environment Details · p. 15

Task Details

  • The tasks cover sorting and stacking boxes, preparing and serving bread in BUDS-Kitchen, arranging a plate and fork, placing a bowl on a plate, and making coffee.Make-Coffee spans opening the k-cup holder through activating the espresso machine; BUDS-Kitchen includes turning off the stove after serving.Source: C Environment Details · p. 15

Data Collection

  • The team collects 100 SpaceMouse-teleoperated demonstrations per simulation task and 50 per real-world task, applying color augmentation to increase visual diversity under lighting variation.The augmentation is intended to improve robustness to visual-cue variations from surrounding lighting conditions.Source: C Environment Details · p. 15

Task Success Determination

  • Success rates use all evaluated rollouts; simulation success requires the simulator’s programmed goal function to return true, while real-world success requires the demonstrated goal configuration.For Sorting, the simulation goal checks whether both boxes are in the bin.Source: C Environment Details · p. 15

Evaluation Horizons

  • Canonical evaluation horizons are 1000 steps for Sorting, 800 for Stacking, and 1500 for BUDS-Kitchen; testing variants receive 200 additional steps per task.Source: C Environment Details · p. 15

Related papers

Loading 2210.11339v2…