PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, Nicolas Heess

arXiv:2306.11706v2cs.ROcs.LG

TL;DR

Real-world robot learning has often focused on one task at a time, while RoboCat is a vision-based generalist manipulation agent for varied robot embodiments. It quickly adapts to downstream tasks using 100–1000 demonstrations, with skill-acquisition costs reported as dramatically lower than single-task baselines.

  • Problem

    Real-world robot learning has often focused on one task at a time, while task design and robot-experience generation are costly.

    Source: §1 · p. 1

  • Method

    RoboCat is a vision-based generalist manipulation agent designed to handle robots with differing degrees of freedom, observation and action specifications, and control frequencies.

    Source: §1 · p. 2

  • Results

    RoboCat quickly adapts to a wide set of downstream tasks using 100–1000 demonstrations, with skill-acquisition costs dramatically lower than single-task baselines.

    Source: §7 · p. 20

  • Takeaways & Limitations

    The paper presents RoboCat as part of a simple self-improvement process.

    Source: §1 · p. 2

    Embodiment transfer between robots can produce undesired movements.

    Source: Broader Impact · p. 21

Abstract

from arXiv · show

The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.

1 Introduction

RoboCat addresses the challenge of leveraging heterogeneous robotic experience by training a multi-task, multi-embodiment transformer agent for visual manipulation. It adapts to unseen tasks and embodiments with 100–1000 demonstrations and can use newly collected experience to improve later iterations.

1 Introduction

  • RoboCat is a large transformer sequence model trained across diverse manipulation behaviours and embodiments, handling different observation and action specifications without common representations.It uses variable-length sequences conditioned on context to handle differences in degrees of freedom and control frequencies.Source: §1 · p. 2
  • RoboCat adapts to unseen tasks, embodiments, behaviours, objects, perceptual variants, and sim-to-real settings using 100–1000 demonstrations.The paper presents this as reducing the cost of acquiring new skills and onboarding embodiments.Source: §1 · p. 2
  • Fine-tuned RoboCat models generate additional experience for later training iterations, improving cross-task transfer, performance on existing tasks, and fine-tuning capabilities.Figure 1 depicts trajectories from newly adapted tasks being added to the training dataset for the next iteration.Source: §1 · p. 2, Figure 1 · p. 3
  • Visual goal-conditioning lets RoboCat specify tasks through images and relabel trajectory frames as hindsight goals, enabling reuse of existing and suboptimal experience without additional human supervision.The agent uses a VQ-GAN encoder pretrained on a broad image set.Source: §1 · p. 2
  • Scaling and broadening RoboCat’s training data improves its performance on training tasks and makes it more efficient at fine-tuning.Source: §1 · p. 2

2 RoboCat

RoboCat is a self-improving generalist agent for multiple manipulation tasks and embodiments in simulation and the real world, trained through a cycle that adds autonomously collected data to future training.

2 RoboCat

  • RoboCat performs multiple tasks across embodiments in simulation and the real world.Source: §2 · p. 3
  • The training cycle pre-trains a VQ-GAN tokeniser, trains a generalist on diverse data, fine-tunes it with human demonstrations, then adds autonomous on-policy data for the next model.The generalist’s training data spans domains and embodiments, and tasks are specified through visual goals.Source: §2 · p. 3

2.1 Training and task specification

  • Tasks are defined by valid start and end states, and success is determined from the final state for tasks whose success can be judged from that state alone.For example, success for “Insert the apple into the bowl” means the apple is inside the bowl.Source: §2.1 · p. 3
  • The goal-conditioned policy selects actions from observations and a goal image that exemplifies the task rather than specifying an exact state to reach.The agent is evaluated for task success, not for reaching the particular state shown in the goal image.Source: §2.1 · pp. 3–4
  • The policy is modeled with an autoregressive transformer, while action and proprioception dimensions vary across embodiments and the model uses tokenized inputs and outputs.Source: §2.1 · p. 4

of tokenised trajectories ˆD = {ˆτ i}|D|

RoboCat trains on goal-augmented tokenised trajectories, using hindsight or semantically equivalent goals, and combines frozen VQ-GAN image tokenisation with action and future-image prediction. The section also describes fine-tuning the generalist on 100–1000 demonstrations per task.

of tokenised trajectories ˆD = {ˆτ i}|D|

  • A tokenised trajectory includes proprioceptive, image, goal, and action tokens across T transitions, with goal observations repeated at each step and some token counts varying by embodiment.L and Q vary by embodiment; the supplied passages identify them as the proprioceptive and action token counts, respectively, while M and N count image and goal tokens.Source: of tokenised trajectories ˆD = {ˆτ i}|D| · p. 4

same task, gi

  • Successful episodes use both hindsight goals from their own final image and semantically equivalent goals from another successful episode; unsuccessful episodes use hindsight goals only.The selected goals are task-conditioned, with success determined by a success detector or reward function.Source: of tokenised trajectories ˆD = {ˆτ i}|D| · p. 4, same task, gi · p. 4

2.1.1 Architecture and pretraining

  • RoboCat uses the Gato transformer architecture and a pretrained, frozen VQ-GAN encoder to tokenize images in advance, while following Gato’s procedure for proprioception and actions.The encoder converts images into latent vectors discretized by nearest-neighbour codebook lookup; diverse-image pretraining is reported to improve generalisation.Source: §2.1.1 · p. 4
  • Adding future-image-token prediction to the standard token-prediction loss improves performance; RoboCat predicts image tokens 5 time steps ahead because adjacent images can be similar.The training data jointly collects trajectories from all training tasks.Source: §2.1.1 · p. 5

2.1.2 Fine-tuning and self-improvement

  • For fine-tuning and self-improvement, the generalist is fine-tuned per task on 100–1000 teleoperated demonstrations, tokenised and goal-augmented as in generalist training.The paper identifies this small-demonstration adaptation as a key contribution, particularly for real robotics where data collection is bottlenecked by robot operation time.Source: §2.1.2 · p. 5

perform the optimisation θy

RoboCat fine-tunes pretrained weights on task demonstrations, then uses deployed policies to collect and relabel trajectories for iterative generalist training. Real-world collection is supported by learned success detection and policy pools that automate resets.

perform the optimisation θy

  • Fine-tuning on task demonstrations starts from pretrained weights, specialises the agent to the new task, and may reduce performance on original tasks.The task-specific demonstration data is used for fine-tuning.Source: perform the optimisation θy · p. 5

section. The resulting relabelled trajectories form a self-improvement dataset Dy

2.2 Real-world deployment

  • For real-world collection, vision-based reward models classify task success from images using binary annotations of demonstrations and policy episodes.Automated success detection supports hindsight relabeling and helps determine when a reset is needed.Source: §2.2 · p. 6
  • Policy pools use tasks with overlapping end and start states as reset mechanisms, automating resets while increasing robot utilisation and initial-condition diversity.Source: §2.2 · p. 6

3 Tasks and Data

RoboCat’s task and data design combines dexterous multi-object manipulation across varied real and simulated embodiments with expert, teleoperated, and self-generated experience.

3 Tasks and Data

  • RoboCat targets diverse, dexterous behaviours requiring fine motor skills, hand-eye coordination, complex affordances, and multi-object interactions.Its data spans simulated and real embodiments and is generated through RL experts, human teleoperation, and RoboCat itself.Source: §3 · p. 7

3.1 Embodiments

  • RoboCat controlled Sawyer arms at 7-DoF in simulation and 5-DoF in reality, Panda arms at 7-DoF in both, and 14-DoF KUKA arms seen only during fine-tuning, across 36 real robots.The 36 real robots comprised 15 Panda, 17 Sawyer, and 4 KUKA arms.Source: §3.1 · p. 7

3.2 Object sets

study types of robotic manipulation that involve multi-object interaction, specifically, structure-building

  • The NIST-i gear set introduces a systematic insertion-affordance study, requiring specific gear-to-peg allocation and meshing within a 1 mm tolerance.The real base is unfixed, increasing task difficulty; the simulated base is fixed.Source: study types of robotic manipulation… · p. 8

3.3 Task families

  • The 253 task variations span structure building, lifting specified objects, and insertion or removal, with variants combining embodiments, object sets, and perceptual goals.Structure building requires precise stable placement; insertion tasks can require low-tolerance positioning, shape understanding, and reorientation.Source: §3.3 · p. 9

3.4 Data sources

  • RoboCat trains on specialist RL trajectories, human-teleoperated expert data, and self-generated data collected during evaluations for reuse in later training.Self-generated data comes from agents fine-tuned on teleoperated expert data and therefore resembles teleoperation behaviours.Source: §3.4 · p. 9

4 Experimental Setup

The experimental setup spans diverse simulated and real robotic tasks, uses limited demonstrations for fine-tuning, and evaluates RoboCat against vision-model and robotics baselines. It also establishes protocols for self-improvement and reproducible evaluation.

4.1 RoboCat training tasks

  • RoboCat trains on 240 tasks and fine-tunes on 13, spanning 2 simulated and 3 real embodiments, 5 simulated and 11 real task families, and 123 simulated and 134 real objects.Source: §4.1 · p. 10
  • RoboCat can self-improve by fine-tuning a previous generalist on new tasks and generating additional data for the next training iteration.These self-improvement tasks were unseen in the previous RoboCat iteration.Source: §4.1 · p. 10

4.2 Training and fine-tuning

  • Most results use a 1.18B-parameter decoder-only transformer; 364M-parameter models support selected ablations.The main model has 24 layers, a 2048 embedding size, and an 8196 post-attention feedforward hidden size.Source: §4.2 · p. 10
  • Real-task fine-tuning uses between 100 and 1000 human teleoperation demonstrations per task.Source: §4.2 · p. 10

4.3 Evaluation

  • Models are evaluated over 100 or more episodes per task, with a different goal image and randomized initial state for each episode.Source: §4.3 · p. 11
  • For real-task fine-tuning, checkpoints are assessed every 5000 steps over 25 episodes, then the best checkpoint is evaluated over 100 episodes.This protocol addresses the lack of a reliable offline measure for choosing fine-tuning duration.Source: §4.3 · p. 11

4.4 Baselines

  • Among 59 evaluated vision-foundation-model baselines, NFNet-f6 (438M parameters) and Swin-L (197M) were selected as main baselines; both use CLIP pretraining.They were smaller than RoboCat because they handle one task and were adapted from existing vision-model architectures.Source: §4.4 · p. 11
  • Vision-model baselines use the same behavioural-cloning loss and successful episodes as RoboCat; other comparisons include Gato on RGB-Stacking and blue-on-green stacking, plus BC-IMP agents for RGB-Stacking.The vision-foundation-model comparison contextualizes task difficulty and the utility of robotics data versus vision datasets.Source: §4.4 · p. 11

5 Experiments

The experiments assess RoboCat’s performance across heterogeneous tasks and embodiments and its adaptation to new scenarios. On training tasks, RoboCat outperforms single-task baselines on most tasks, with a pronounced advantage in the real world.

5 Experiments

  • The evaluations ask whether RoboCat learns from heterogeneous data to solve visual-goal tasks across dexterous physical and simulated embodiments.Source: §5 · p. 11
  • RoboCat performs better than single-task agents trained on the same data on the vast majority of training tasks.The comparison uses the best-performing visual foundation model-based agents selected from 59 baselines.Source: Figure 5 · p. 12
  • In real-world tests, RoboCat’s multi-task joint training on robotics data performs significantly better than the baselines.Source: Figure 5 · p. 12
  • A much smaller RoboCat performs similarly on simulated stacking tasks but achieves much lower success on more challenging cases.Source: Figure 5 · p. 12
  • The evaluations also ask whether RoboCat can adapt from a small number of demonstrations to unseen tasks, objects, and embodiments with new morphology and action spaces.Source: §5 · p. 12

3. Does RoboCat exhibit cross-task transfer and generalisation to held-out tasks? (Section 5.2)

This section examines RoboCat’s generalisation to held-out tasks and its adaptation using limited demonstrations, including across objects and embodiments. It also covers self-improvement through generated data, along with related ablations and appendix evaluations.

5.2 Generalisation and adaptation

  • Across 23 held-out tasks, RoboCat-lim generalised zero-shot in simulation to a held-out object set on Sawyer and blue-on-green stacking on Panda, but not that variant on Sawyer.It fine-tuned to that task variant with as little as 100 demonstrations; on the real-world variant, it achieved 88% with 500 demonstrations.Source: §5.2 · p. 15
  • RoboCat-lim adapted with 100–500 episodes to unseen variations and objects, different data sources, and an unseen task on an embodiment with twice as many degrees of freedom as training embodiments.The real-world tasks included stacking and tower building, inverted pyramid building, and dexterous KUKA manipulation; the inverted pyramid task had only 52% teleoperator success, while KUKA success was nearly 80%.Source: §5.2 · p. 15
  • On the unseen KUKA embodiment, zero-shot success was 0%, but fine-tuning with a relatively small amount of embodiment-specific data yielded 69% success.The paper attributes the zero-shot result to KUKA’s entirely different action and observation spaces.Source: §5.2 · p. 15
  • On the simulated few-shot tasks compared, RoboCat-lim learned with significantly less data than the visual foundation model baselines.Figure 8 reports results for single task variants and shows RoboCat-lim outperforming baselines with the same number of episodes, including on a task where its zero-shot success was 0%.Source: §5.2 · p. 15, Figure 8 · p. 16
  • RoboCat showed positive transfer: the broader-trained model outperformed RoboCat-lim on its training tasks and adapted better to real-world insertion and removal.The paper notes that training on real-world fruit and vegetable lifting data helped RoboCat adapt to insertion and removal with fruits and a bowl.Source: §5.2 · pp. 15–16, Figure 9 · p. 16

5.3 Self-improvement via RoboCat fine-tuning and self-generation of data

  • The self-improved agent outperformed the direct-demonstration baseline on all four tasks.The procedure combined fine-tuning on task demonstrations with self-generation of additional data.Source: §5.3 · p. 17
  • For most cases, the full RoboCat generalist performed similarly to or better than the agents that generated its data.The scaled process incorporated self-generated data from numerous task-specific fine-tuned agents; the resulting agent showed better generalisation and fine-tuning capabilities across a broader set of real-world tasks.Source: §5.3 · p. 17

5.4 Further ablations

  • Further appendix evaluations covered tokenizer and observation-prediction choices, model sizes, vision-model baselines, and NIST-i environment ablations.The section points to Appendices D.3 and G.2–G.4 for these analyses.Source: §5.4 · p. 18

6 Related Work

Related work frames RoboCat’s contribution as broader multi-embodiment robotic control, direct joint action prediction, and a self-improvement loop, while clarifying its visual-goal assumptions.

6.1 Transformers for decision making

  • Unlike VIMA (Jiang et al., 2022), which uses high-level spaces and predefined action primitives, RoboCat extends Gato (Reed et al., 2022) toward broader robotic task variety and scale, with positive transfer and fast real-world adaptation.Source: §6.1 · p. 18

6.2 Visual pretraining for control

  • RoboCat jointly predicts actions across embodiments rather than planning with video prediction, using a frozen pretrained VQ-GAN to speed training.Source: §6.2 · p. 19

6.3 Goal-conditioned policies

  • RoboCat uses goal images as task specifications for behavior cloning and assumes goal-image datasets during evaluation and collection, leaving goal distance, generation, and exploration unaddressed.Source: §6.3 · p. 19

6.4 Generalist robotic agents

  • RoboCat natively supports real-world embodiments with differing observations and actions and self-improves by fine-tuning on new tasks and generating retraining data.Its scope spans simulation and real robots, including precision, dexterity, and low-level control. The work focuses on visual goal conditioning, while language or full-demonstration conditioning are future possibilities.Source: §6.4 · p. 19

7 Summary and Future Work

RoboCat generalizes across tasks, embodiments, and settings, adapts from small demonstration sets, and can improve through data generated by adapted agents. The paper also identifies future research directions and safety considerations for real-world robotic agents.

7 Summary and Future Work

  • RoboCat solves a large and diverse set of goal-image tasks across task families, embodiments, control modes, and objects in simulation and the real world, and adapts via fine-tuning on 100–1000 demonstrations.Adapted agents can generate data added to RoboCat’s training dataset for future iterations, a process called self-improvement.Source: §7 · p. 20
  • Scaling and diversifying training data improved performance on training tasks and adaptation to unseen tasks, while skill acquisition cost was lower than for single-task baselines, including visual-foundation-model baselines.The report describes extensive capability evaluations in simulation and on real robots.Source: §7 · p. 20
  • The final agent adapted to 13 unseen downstream tasks, with 22 more in a generalization study and nine in self-improvement, spanning new embodiments, task families, perceptions, real-world tasks, and objects.Examples include a KUKA with a dexterous hand, lifting and insertion/removal, held-out perceptual variations, real versions of seen simulation tasks, and printed fruits and vegetables.Source: §7 · p. 20
  • Future work could combine visual goals with language task specification and investigate reinforcement learning for reward-based learning and online real-world interaction.RoboCat currently uses behavior cloning; language may better specify some tasks and language-conditioned LLM/VLM baselines may offer stronger temporal reasoning.Source: §7 · p. 20
  • The authors also call for robustness beyond controlled labs with visually similar backgrounds, including operation amid more diverse environments and basket textures.Visual goals let the agent learn from failures and suboptimal data; reinforcement learning could additionally support learning from rewards and online interaction.Source: §7 · p. 20

Broader Impact

  • RoboCat is for research use only and is not deployed in production; physical actions and cross-embodiment transfer raise safety risks, including misplaced trust and undesired movements.The paper also notes the need for interdisciplinary work on risks and benefits of generalist robotic agents.Source: Broader Impact · p. 21
  • The self-improvement loop creates additional safety concerns, but data collection is human-controlled and learning uses frozen agents and fixed data, decoupled from collection.The authors call for further work on safety for reinforcement-learning robotic systems and adapting alignment techniques to this setting.Source: Broader Impact · p. 21

RoboCat generalist training … B.1.4 NIST-i tasks

The appendix describes RoboCat’s training tasks and data, cataloguing 253 task variations and detailing lifting, building, insertion/removal, and NIST-i task designs.

A Model Card

B Tasks and Data

  • Section B describes the tasks and data used to train RoboCat.It introduces an extensive description of the training tasks and data.Source: B Tasks and Data · p. 31

B.1 Task families

B.1.1 Lifting objects

  • Lifting tasks study goal understanding and generalisation across tasks and embodiments, using physical-world and simulated objects, including NIST-i gears.The family includes seven NIST-i gear variants, with lifting in simulation and on the real Panda 7DoF; the large gear is also lifted with the KUKA 14DoF.Source: §B.1.1 · p. 31

B.1.2 Building structures

  • Building tasks include stacking, tower, pyramid, and inverted-pyramid arrangements, where non-cubic objects and precise placement make stable construction challenging.Some difficult tower-building variations were excluded from data collection; the tasks span simulation and real-world settings, though pyramid building is simulated only and inverted-pyramid building is real-world only.Source: §B.1.2 · pp. 31, 34

B.1.3 Inserting and removing objects

  • Insertion/removal tasks cover fruit in a bowl, shape-matching objects on a board, and NIST-i gears on shafts, which require precise fitting under tight tolerances.Shape matching also requires identifying the correct opening and reorienting the object; the board can move, while real NIST-i shaft bases are unanchored.Source: §B.1.3 · p. 34

B.1.4 NIST-i tasks

  • Compared with the official NIST benchmark, NIST-i has a movable base, red gears and a black board, and looser insertion tolerance.The physical NIST-i insertion tolerance is 1 mm, versus sub-millimeter tolerance for the original NIST board.Source: §B.1.4 · p. 34
  • Simulation and physical NIST-i tasks differ in color; red-only gears made simulated stacking teleoperation difficult, while RGB colors improved demonstrator contrast.For simulated insertion/removal, the shaft base was fixed to make tasks easier for raters and agents and to speed up simulation.Source: §B.1.4 · p. 35

B.2 Data sources … D.2 Model architecture and loss

This section describes RoboCat’s data sources and robot embodiments, then details the KUKA setup, episode resets, and VQ-GAN training data, architecture, and loss.

B.2 Data sources

  • Simulated structure-building data came from task-specific off-policy RL policies with shaped rewards; state-based generation was faster than vision, and privileged object poses were discarded for RoboCat training.The authors used MPO, while noting that any RL algorithm could have been used.Source: §B.2 · p. 35
  • Across 4 countries, researchers collected over 4000 hours of human demonstrations across many tasks; teleoperators saw different language instructions each episode, and “shuffle” episodes diversified starts and improved agent robustness.Source: §B.2 · p. 35
  • After collecting a minimum of successful human episodes, the team fine-tuned RoboCat on 500 or 1,000 successes, used it to autonomously gather more data, then trained the final agent on generated data and demonstrations.The self-generation process was applied to the same task variation used for fine-tuning.Source: §B.2 · p. 36

C Embodiments and Environments

  • Experiments used Sawyer, Panda, and KUKA arms with different controllers and end-effectors; despite sharing degrees of freedom, their dynamics differed, and simulation and reality were not aligned.Simulation used physics randomization but no visual randomization.Source: C Embodiments and Environments · p. 36

C.1.1 Sawyer

  • Sawyer used a 7-DoF action space in simulation but a simplified 5-DoF real-robot space that restricted gripper orientation to vertical.The real action space comprised 3D translation, 1D vertical-axis rotation, and gripper velocity control.Source: §C.1.1 · p. 39

C.1.2 Panda

  • Panda used a 7-DoF action space in simulation and reality, but its simulated Cartesian controller built on joint-velocity control while the real controller tracked velocity through joint torque.Two gripper-mounted cameras were not used to train or evaluate RoboCat.Source: §C.1.2 · p. 39

C.1.3 KUKA

  • KUKA was used only in the real world with a 14-DoF action space: 6-DoF Cartesian velocity control plus an 8-DoF hand command; its gripper cameras were unused.The arm was paired with a three-fingered hand.Source: §C.1.3 · p. 40
  • The KUKA observations include arm joint positions, velocities and torques, estimated wrist forces and torques, tracked TCP reference pose, and three-finger-hand proprioception.Only a subset of proprioception observations is used for RoboCat training.Source: §C.1.3 · p. 40

C.2 Basket and setup

  • All embodiments use a standardized cage and 25 × 25 cm basket workspace with two front-corner Basler ace cameras; Sawyer images are 128 × 128, versus 296 × 222 for Panda and KUKA.The basket defines the robot arm’s workspace (Lee et al., 2021).Source: §C.2 · p. 40

C.3 Environment resets

  • In real environments, objects generally remain between episodes while the arm is moved to a new random pose; RGB objects instead receive scripted pick-and-place resets using a blob-based 3D tracker.The RGB reset uses the RGB Stacking benchmark procedure (Lee et al., 2021); simulation places objects in new randomized poses each episode.Source: §C.3 · p. 40
  • In simulation, objects are placed in new randomized poses at the start of each episode.Source: §C.3 · p. 40

D.1 Datasets

  • The RoboCat-lim and RoboCat VQ-GANs use distinct image-data mixtures, with the latter adding simulated YCB fruit lifting, NIST-i gear images, and real Sawyer data.RoboCat-lim uses ImageNet, DeepMind Control Suite, Meta-World, domain-randomized simulated Sawyer stacking, simulated Panda, and real Sawyer red-on-blue stacking images.Source: §D.1 · p. 41

D.2 Model architecture and loss

  • The VQ-GAN combines a ResNet encoder and decoder with Transformer attention and uses 64 quantizer embeddings of dimension 128, with weighted discretization and reconstruction losses.The loss is 0.25 × discretisation_loss + l2_reconstruction_loss + 0.1 × log_laplace_loss; training uses batch size 64 for 1 million steps.Source: §D.2 · p. 41

D.3 Ablations for our VQ-GAN design choices … E.3 Training and fine-tuning parameters

The ablations favor VQ-GAN tokenisation and future-token prediction for generalisation, with a five-step prediction horizon performing best. The appendices specify the task inputs, image processing, data weighting, and training and fine-tuning settings.

D.3 Ablations for our VQ-GAN design choices

  • Patch ResNet performs better on most training families but worse on held-out tasks, while VQ-GAN generalises better across scales.The VQ-GAN tokeniser’s advantage requires diverse training data including ImageNet and observation-token prediction.Source: §D.3 · pp. 41, 45, Figure 18 · p. 45
  • Pixel prediction leaves training performance similar but yields zero generalisation success, whereas future-token prediction performs better than future-pixel prediction.The pixel-prediction policy appears to ignore the goal and perform a different task.Source: Figure 19 · p. 45, §D.3 · pp. 45–46
  • Increasing the prediction horizon from k = 0 to k = 5 improves generalisation by 50% relative, while longer horizons degrade performance; k = 5 is optimal.Training performance rises with the horizon but shows diminishing returns beyond k = 5.Source: Figure 19 · p. 45, §D.3 · p. 46
  • Increasing VQ-GAN vocabulary size leaves training performance unchanged but monotonically reduces generalisation ability.Source: Figure 19 · p. 45, §D.3 · p. 46

The training data for each task contains

Appendix C.2).

  • Images are cropped and resized to 128 × 128 × 3, augmented by brightness, saturation, contrast, and translation changes, and trajectories are trimmed when images stop changing significantly.Source: Appendix C.2 · p. 46

E.2 Data weighting

  • RoboCat-lim samples task variants, failures, and successes equally, uses hindsight and equivalent successful-episode goals equally, and limited-demo fine-tuning uses only successes.Source: §E.2 · p. 47

E.3 Training and fine-tuning parameters

  • Pretraining uses AdamW with cosine decay by a factor of 10 over 2,000,000 steps, batch size 256, and sequence length 1,024; fine-tuning uses Adam at 1e−5 for up to 50,000 gradient steps.Dropout rate is 0.1.Source: §E.3 · p. 47

E.4 RoboCat-lim training data … G.4.1 Task complexity

This section describes RoboCat-lim’s training and held-out task design and evaluation procedures across simulation and real-world settings, including success detection and policy pools for tasks without scripted rewards or resets. It then reports evaluation findings on performance, model size, task conditions, and data, including the smaller agent’s stronger performance on fixed-base NIST-i, which the authors suggest may reflect the setting’s similarity to simulation and positive skill transfer.

E.4 RoboCat-lim training data

  • Training combines simulated Panda and Sawyer structure-building tasks with real Sawyer stacking, while holding out perceptual variants, objects, task families, and embodiments for evaluation.Real blue-on-green stacking and Sawyer stacking with the RGB triplet 5 are held out; real Panda tasks test sim-to-real transfer, inverted pyramid building tests an unseen task family, and KUKA 14DoF tests embodiment transfer.Source: §E.4 · pp. 47–48

F Evaluation and Real-World Deployment

  • Evaluations use at least 100 episodes per task, shorter 25-episode checks for fine-tuned checkpoints, randomized initial conditions, and task-matched control frequencies and episode lengths.The best fine-tuned checkpoint is selected before its 100-episode evaluation.Source: F Evaluation and Real-World Deployment · pp. 48–49

F.1 Evaluation for real-world RGB tasks

  • For real-world RGB tasks, visual counting replaces the inherited reward criterion because its arm-height thresholds miss clear stacking successes; prior evaluations were also visually recounted.The procedure follows Lee et al. (2021) infrastructure and scripted resets, apart from the reward function.Source: §F.1 · p. 49

F.2 Evaluation for real-world tasks with no rewards or scripted resets

  • For other real-world tasks, evaluation omits scripted rewards and reset policies, using human-annotation-trained success detectors and a policy pool in which tasks can reset one another.This simplifies task implementation but adds evaluation complexity.Source: §F.2 · p. 49

F.2.1 Success detection

  • Fewer than five annotations typically label a 1,200-step episode by marking solved-state transitions and extending labels between them; detectors are not used to report performance because accuracy is around 90%.Reported successes are visually assessed.Source: §F.2.1 · pp. 49–50

F.2.2 Policy pools as generalised reset policies

  • Policy pools provide resets, diversify evaluation initial conditions, and interleave tasks to use robot time efficiently; larger, more diverse pools generate more varied starts.The scheduler selects an eligible task based on the environment state.Source: §F.2.2 · pp. 50–51
  • A policy pool can support single-task evaluation with a different policy per task or multi-task evaluation with one shared policy; task-to-policy mappings need not be one-to-one.For RoboCat evaluation, the pool uses RoboCat’s policy for every task.Source: §F.2.2 · p. 51

G.1.1 Training tasks

  • On simulated structure-building tasks, RoboCat’s performance is within 10% of state-based experts except for tower building; on real stacking it exceeds data-generating agents.The experts used privileged state information, unlike the vision-based RoboCat agent.Source: §G.1.1 · p. 51, Figure 20 · p. 53
  • On average, RoboCat matches human teleoperator performance on the NIST-i tasks in both simulation and the real world.Human-teleoperated demonstrations were used for these tasks.Source: §G.1.1 · p. 51, Figure 22 · p. 54, Figure 23 · p. 55

RB RG GR GB BR BG

  • RoboCat on average matches the training data’s success rate and exceeds it in some cases, including lifting banana, peach, strawberry, and potato.The training data includes successful self-generated episodes and successful human teleoperations.Source: RB RG GR GB BR BG · p. 53, Figure 24 · p. 55, Figure 25 · p. 56

G.1.2 Fine-tuning performance

  • More fine-tuning demonstrations generally improve performance, sometimes bringing it near human-operator levels, but insert tasks remain difficult because they require reorientation and shape understanding.Figure 28 presents final performance for each fine-tuning task.Source: §G.1.2 · p. 53, Figure 28 · p. 57

G.2 Impact of model size

  • The 1.18B-parameter model improves success rates by 6–21% over the 364M model across simulated structure-building task families, with larger gaps on pyramid and tower-building tasks.The smaller model only slightly degrades performance on the easier stacking tasks.Source: §G.2 · p. 53, Figure 26 · p. 56

G.3 Visual foundation baselines

  • Among pretrained visual-model baselines, non-frozen models performed much better than frozen ones, so only non-frozen results are reported.The baselines were evaluated on training and held-out tasks with limited demonstrations.Source: §G.3 · p. 58, Table 17 · p. 58

G.4 NIST-i extended study

  • In NIST-i studies, fixing the physical base, using more cameras, and adding data from largely unrelated tasks are reported to improve performance; a moving base is argued to make physical tasks harder.The unrelated-task result is described as suggesting positive skill transfer.Source: §G.4 · p. 59, §G.4.1 · p. 59

G.4.1 Task complexity

  • A 364M agent trained on RoboCat-lim’s data performs much better on the fixed-base NIST-i task than on the other two tasks, despite no physical fixed-base training data.Table 18 reports average success over all three NIST-i gear sizes.Source: Table 18 · p. 59, §G.4.1 · p. 59
  • The authors suggest the fixed-base environment may be easier because it resembles simulation, enabling positive skill transfer.Source: §G.4.1 · p. 59

G.4.2 Data bias · G.4.3 Skill transfer · G.4.4 Additional observations

Specialist behavior-cloning results show no evidence of NIST-i data bias toward a centered, simulation-like peg base, while full RoboCat reduces gear-insertion failures in both base settings. Additional observations show that wrist-camera inputs improve specialist performance, motivating further study of camera observations for contact-rich manipulation.

G.4.2 Data bias

  • To test whether apparent transfer reflected data bias, the authors trained task-specific Specialist BC agents using only each NIST-i task’s training subset.Because behavior-cloning agents can overfit their training data, the authors expected a bias-driven advantage on fixed-base tasks if demonstrations favored a centered, simulation-like peg base.Source: §G.4.2 · p. 59
  • Specialist BC agents performed similarly on fixed- and moving-base tasks, providing no evidence of bias toward a centered, simulation-like peg base.The relationship between fixed- and moving-base performance was inverted, even though all training data was collected with a moving base.Source: Table 19 · p. 59, §G.4.2 · p. 60

G.4.3 Skill transfer

  • Moving from the 364M model to the full RoboCat agent eliminates just under two thirds of gear-insertion failures in both fixed- and moving-base settings, indicating a smaller performance gap.Source: Table 20 · p. 60
  • The full RoboCat agent’s smaller performance gap between base states suggests positive skill transfer to moving-base insertion tasks.The models use the same NIST-i insertion data, while full RoboCat has greater capacity and is trained on more tasks.Source: §G.4.3 · p. 60

G.4.4 Additional observations

  • Four camera observations—two basket views and two wrist views—significantly improved Specialist BC performance compared with using two camera observations.The paper identifies the effect of different camera observations for contact-rich manipulation with RoboCat as a future research direction.Source: §G.4.4 · p. 60, Table 21 · p. 60
  • The authors identify studying different camera observations for contact-rich manipulation with RoboCat as a future research direction.The observation was prompted by the benefit of adding wrist cameras for specialist agents.Source: §G.4.4 · p. 60

Related papers

Loading 2306.11706v2…