PaperlayerView exact revision on arXiv

Source-linked AI summary

Visionary: Vision architecture discovery for robot learning

Iretiayo Akinola, Anelia Angelova, Yao Lu, Yevgen Chebotar, Dmitry Kalashnikov, Jacob Varley, Julian Ibarz, Michael S. Ryoo

arXiv:2103.14633v1cs.ROcs.CVcs.LGcs.NE

TL;DR

Robot-learning systems must choose how to combine visual features with low-dimensional actions, while task-specific architecture design is challenging in sparse-reward reinforcement learning. Visionary jointly searches vision-action architectures and trains them for manipulation. The paper reports a large success-rate gain in simulated stacking and better performance than the baseline in real-robot grasping.

  • Problem

    Robot learning still faces the challenge of choosing suitable visual architectures and how to connect low-dimensional actions with high-dimensional visual features for complex tasks.

    Source: §I · p. 1

  • Method

    Visionary jointly and differentiably searches vision-action architecture connectivity while training the model for the robot manipulation task.

    Source: §I · pp. 1–2

  • Results

    Visionary outperforms prior work by a large jump in task success rate on simulated block stacking and performs better than the baseline on real-robot grasping.

    Source: A. Quantitative performance evaluation… · p. 6, B. On-Robot Grasping Experiments · p. 6

  • Takeaways & Limitations

    The paper reports meaningful improvements in simulation and real-robot tasks from developing the architecture alongside training for the robot task.

    Source: §VI · p. 6

Abstract

from arXiv · show

We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs. Our approach automatically designs architectures while training on the task - discovering novel ways of combining and attending image feature representations with actions as well as features from previous layers. The obtained new architectures demonstrate better task success rates, in some cases with a large margin, compared to a recent high performing baseline. Our real robot experiments also confirm that it improves grasping performance by 6%. This is the first approach to demonstrate a successful neural architecture search and attention connectivity search for a real-robot task.

I. INTRODUCTION

Robotic manipulation requires visual architectures that can adapt to complex tasks and combine visual features with low-dimensional actions, a choice that existing systems often leave hand-designed. Visionary jointly searches architecture connectivity and trains the policy for the task, including real-robot grasping.

I. INTRODUCTION

  • Robotic tasks require architectures suited to complex tasks and a way to combine low-dimensional actions with high-dimensional visual features, a choice prior systems often handle manually.The paper also notes challenges in adaptability, task complexity, and sample complexity.Source: §I · p. 1
  • Visionary jointly learns model architecture and trains the model for the task, using one-shot connectivity optimization in reinforcement learning.The approach addresses sparse rewards, evolving data, and the expense of repeatedly modifying architectures within the RL loop.Source: §I · p. 1
  • Prior neural architecture search methods target supervised vision tasks and can be computationally costly, while Visionary searches action–vision combinations for RL-based robot learning.The cited prior approaches are described as lacking action-input vision-control loops and as costly to combine with sparse-reward exploration.Source: §I · p. 2
  • Visionary discovers architectures with learned combinations of robot actions and visual inputs, optimized for robotics tasks including real-robot grasping.The paper presents a differentiable search that learns to combine vision and action inputs while training the network.Source: §I · p. 2
  • The approach is presented as general and flexible for tasks that require vision and action inputs.The contribution also identifies sparse-reward exploration and computational feasibility as robotics challenges addressed by the search.Source: §I · p. 2
  • The authors report performance gains from architecture search on a real-robot task and identify peer-attention as a way to learn connectivity between visual and action inputs.The discovered task-specific architectures outperform previously known architectures, according to the paper.Source: §I · p. 2

II. RELATED WORK

Prior vision-based robot learning has used deep networks for tasks such as grasping, while action inputs are often merged with visual features in an ad hoc way. Visionary instead explores task-guided combinations and uses differentiable architecture search for robotic manipulation.

II. RELATED WORK

  • Deep learning-based perception and end-to-end systems have been applied to robot tasks including grasping, object search, and raw-pixel grasping.QT-Opt is described as learning sophisticated grasping skills from raw pixels.Source: §II · p. 2
  • Earlier robot-learning architectures commonly combine action vectors with visual embeddings through standard networks, often using ad hoc concatenation.Visionary directly explores alternative functional transformations and combinations guided by the robot-learning task.Source: §II · p. 2
  • Neural architecture search includes evolutionary, differentiable, and reinforcement-learning approaches; this work uses differentiable search to obtain a Q-learning model for robotic manipulation.The paper also contrasts its approach with QT-Opt by adapting architecture learning alongside grasp-skill learning.Source: §II · p. 2

III. PRELIMINARIES

The paper frames manipulation as reinforcement learning and focuses on discovering neural architectures for multimodal Q-functions. Its experiments use simulated and real grasping, with image, robot-state, and action inputs.

III. PRELIMINARIES

  • The reinforcement-learning formulation models manipulation as a Markov Decision Process in which an agent observes states, takes actions, and seeks to maximize expected discounted cumulative reward.The reward function captures desired behavior.Source: §III · p. 2
  • The method uses QT-Opt, a continuous-action Q-learning variant, to learn a state-action value function and derive actions by optimizing the learned Q-function.The policy is obtained using the cross-entropy method.Source: §III · p. 2
  • The architecture-search focus is to discover more expressive Q-functions by wiring multimodal observations with components of different dimensions.The paper introduces an algorithm for finding task-specific connections in these multimodal Q-functions.Source: §III · p. 2
  • The supernet fusion module learns weights over candidate action-merging operations, then retains the operation with the largest weight for computational efficiency.The figure contrasts this learned fusion with conventional addition-based merging.Source: Figure 2 · p. 3
  • Experiments use a PyBullet-based simulator and validate grasping with real hardware.The learning platform is adapted from QT-Opt.Source: §III · p. 3
  • The task observations include a 472×472 RGB image, gripper state, and height; actions encode pose displacement and a gripper command, with sparse success rewards and a QT-Opt baseline.The success reward is 1.0 and each step carries a -0.01 penalty; the baseline merges action by addition after the fifth convolutional layer.Source: §III · p. 3

IV. APPROACH

Visionary jointly learns a Q-value network’s architecture and weights for the task. It replaces costly nested architecture optimization with a differentiable one-shot search during RL training.

A. Robot architecture learning formulation

  • The method jointly optimizes the Q-value network’s architecture and CNN weights to improve performance on the task.The architecture parameters govern choices such as layers and connections, while CNN parameters include convolution-filter values.Source: A. Robot architecture learning… · p. 3
  • Direct bi-level optimization is expensive because each outer-loop step requires inner-loop RL training and evaluation.Black-box search also faces noisy fitness scores from randomness in inner-loop RL training.Source: A. Robot architecture learning… · p. 3
  • One-shot differentiable joint optimization combines the original inner and outer loops into a single loop, with gradients flowing end-to-end.The formulation makes architecture parameters differentiable with respect to the training loss.Source: A. Robot architecture learning… · p. 3
  • Architecture parameters are regressed jointly with CNN filter values during RL training, allowing the model to replace an existing model in an RL framework.The approach is described as requiring replacement of the previous model with the one-shot discovery model.Source: A. Robot architecture learning… · pp. 3–4

B. Architecture search space

The search space centers on ways to combine low-dimensional actions with visual representations. It searches both the merging operation and its location, as well as attention connections between representations.

B. Architecture search space

  • The search asks how and where to merge the action vector with the high-dimensional visual state representation.The action and visual inputs differ in shape and semantics.Source: B. Architecture search space · p. 4
  • Candidate action-merging operations include addition, concatenation, Hadamard product, and generating convolutional filters from the action.The operations apply a shape-expansion function or learned action-dependent transformations as needed.Source: B. Architecture search space · p. 4
  • Merging may occur once or repeatedly at intermediate visual representations, from early to later CNN layers.The search also includes attention connections between pairs of CNN layers, allowing visual representations to be conditioned on actions.Source: B. Architecture search space · p. 4

C. Differentiable action merging search

A differentiable supernet places candidate action-merging operations into the computation graph and learns their relative weights. It can also learn to omit merging at a location.

C. Differentiable action merging search

  • The supernet includes candidate action-merging operations after computational layers, and learned weights determine which operations matter for a high-performing model.The method excludes the very early layers and includes the candidates in a shared computation graph.Source: C. Differentiable action merging search · p. 4
  • At each layer, the fusion module combines action-merging blocks applied to the visual representation and action vector using learned weights.Each block implements a candidate operation, such as addition or concatenation.Source: C. Differentiable action merging search · p. 4
  • Softmax-constrained weights form a weighted sum of candidate operations; after learning, non-argmax connections are pruned for inference.A no-op candidate lets learning choose not to merge actions at a potential location.Source: C. Differentiable action merging search · p. 4

D. Differentiable attention connectivity search

Visionary searches peer-attention connections so visual representations can be conditioned on actions and other layer representations. Global average pooling makes this channel-wise attention, and the connections add little computation.

D. Differentiable attention connectivity search

  • The final one-shot discovery model combines supernet and peer-attention modules and has a parameter count comparable to the baseline once learned.The model is fully differentiable and trained with RL to discover a CNN architecture optimized for the policy.Source: Figure 4 · p. 5
  • Peer-attention searches connections between CNN layers, enabling visual representations to be conditioned on action representations.The method searches these connections as part of architecture discovery.Source: D. Differentiable attention… · p. 5
  • Global average pooling makes peer-attention channel-wise, distinguishing it from the Hadamard-product action-merging operation.The attention function is applied to representations connected through the module.Source: D. Differentiable attention… · p. 5
  • Peer-attention includes the action representation among its inputs, and its learned connections add 0.35% to total FLOPS.The module uses a weighted summation of one-dimensional channel vectors without spatial resolution.Source: D. Differentiable attention… · p. 5

E. Full one-shot architecture search

Visionary searches over vision–action architectures during training, using fixed offline experience and peer-attention to connect representations. The found model has computation comparable to the conventional baseline.

E. Full one-shot architecture search

  • Architecture search uses off-policy reinforcement learning with a fixed experience dataset sampled continuously from a replay buffer.This supports fair comparison of candidate architectures using the same offline data.Source: E. Full one-shot architecture search · p. 5
  • The comparison figure shows peer-attention connections with weights greater than 0.5 and reports similar computation time to the conventional baseline.Source: Figure 5 · p. 5
  • The one-shot model adds five candidate action-merging locations, peer-attention after each supernet module, and parallel convolutional layers with dilation rates 2 and 4.The parallel layers connect through peer-attention rather than being directly fused with other representations.Source: E. Full one-shot architecture search · p. 5
  • The found model uses 41.05 GFLOPS, comparable to the conventional baseline’s 40.91 GFLOPS.The full search model requires 41.09 GFLOPS.Source: E. Full one-shot architecture search · p. 5

V. EXPERIMENTAL RESULTS

The search model replaces the conventional CNN within the RL framework and is trained with the same hyperparameters as final training. Simulation and real-robot runs use different iteration counts.

V. EXPERIMENTAL RESULTS

  • The one-shot search model replaces the conventional CNN in the standard RL framework, with architecture parameters updated by backpropagation using the RL loss.Source: §V · p. 6
  • Training uses batch size 4096, truncated-normal initialization, L2 regularization of 9e-5, and SGD with momentum 0.958 and learning rate 0.0044.Source: §V · p. 6
  • Models train for 80K iterations in simulation and approximately 110K iterations on the real robot.Source: §V · p. 6

A. Quantitative performance evaluation in simulation

In simulated block stacking, Visionary outperforms the baseline by a large jump in task success rate. Ablations attribute gains to action merging and learned layer connectivity, while the evolutionary-search comparison is less effective.

A. Quantitative performance evaluation in simulation

B. On-Robot Grasping Experiments

On a physical KUKA robot, Visionary performs better than the baseline on random-object grasping from a bin. The conclusion reports meaningful improvements on both simulated and real-robot tasks.

B. On-Robot Grasping Experiments

  • Visionary performs better than the baseline on random-object bin grasping with a KUKA robot, after architecture learning from offline data and testing in a different, more challenging environment.Source: B. On-Robot Grasping Experiments · p. 6

VI. CONCLUSION

  • The conclusion reports meaningful improvements in simulation and on real-robot tasks from differentiable architecture search trained alongside the robot task.Source: §VI · p. 6

Related papers

Loading 2103.14633v1…