Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

• Sparse-reward robotic manipulation can require millions of samples, while imitation learning may require hundreds or thousands of expert demonstrations [18],,.Source: §I · p. 1
• Prior simulated RL work has improved sample efficiency through unsupervised representation learning and data augmentation, motivating their application to real robots.Source: §I · p. 1
• CoDER first stores 10 demonstrations, then contrastively pre-trains the encoder, and finally trains off-policy RL with augmented demonstration and online data.The authors argue that this combination enables efficient real-robot pixel learning, whereas algorithms using the components individually do not.Source: §I · p. 2
• CoDER learns policies for 6 diverse manipulation tasks in 15–50 minutes per task using a lightweight real-robot setup.The setup uses a robot, one GPU, two RGB cameras, a handful of demonstrations, and sparse rewards.Source: §I · p. 2

Soft Actor Critic

• SAC is an off-policy algorithm that learns an action-conditioned value function and a stochastic policy maximizing expected returns.It is used as the base algorithm for sparse-reward manipulation, including pixel inputs with data augmentation.Source: §II · p. 2
• SAC trains its actor to maximize expected return and its critics to match Bellman targets sampled from the replay buffer.The target is an entropy-regularized exponential moving average of the critic ensemble; the action-entropy coefficient affects exploration.Source: §II · p. 2

• The six real-world tasks use concatenated processed views from two cameras, with sparse reward given only when the task is completed.Source: Figure 3 · p. 3

Unsupervised Contrastive Pretraining

• Contrastive learning increases agreement between positive query-key pairs while reducing agreement with negatives; image methods can form pairs from two augmentations of one image.In minibatches, other sampled images serve as negatives in the InfoNCE loss.Source: §II · p. 3

III. METHOD

• CoDER uses three stages: collect demonstrations, pre-train the encoder, then learn a policy with off-policy RL and augmented data.The demonstration-initialized replay buffer supplies both demonstration observations and online observations during RL updates.Source: §III · pp. 3–4
• Ten demonstrations initialize the replay buffer, and real-world collection takes less than 10 minutes including environment resets.The authors found ten demonstrations sufficient for learning skills quickly.Source: §III · p. 3
• The convolutional encoder is pre-trained with instance-based contrastive learning using stochastic random crops to form query-key pairs.The pre-training begins after the replay buffer is initialized with demonstrations.Source: §III · p. 3
• Policy learning starts in around 30 minutes, and as little as 3 minutes for simple tasks such as Reach.The figure distinguishes first task success from learning an optimal policy.Source: Figure 4 · p. 4

A. Experimental Setup

• A position-controlled xArm with a parallel two-jaw gripper uses three degrees of freedom, with actions specifying end-effector and gripper-aperture displacement.Source: A. Experimental Setup · p. 4
• The gripper is confined to a 25 cm-high workspace, and most tasks place objects in a roughly 40 × 34 cm tray.Source: A. Experimental Setup · p. 4
• Two RGB cameras provide an over-the-shoulder arm view and a local object-level view from inside the gripper.Images are cropped and down-sampled to 100 × 100 pixels for the training algorithm.Source: A. Experimental Setup · p. 4
• Joystick demonstrations specify end-effector and gripper-aperture displacement, and collecting each task's demonstrations takes less than 10 minutes.The estimate includes resets and possible human error or inefficiency.Source: A. Experimental Setup · p. 4

B. Environments and Baselines

• The six real-robot tasks are reaching, block pickup, block moving, pulling a large deformable object, flipping a switch, and opening a drawer.Source: B. Environments and Baselines · p. 4
• Reach, pickup, and move adapt OpenAI Gym Fetch tasks for the real robot, while the experiments also include pull, switch flipping, and drawer opening.Pull has prior imitation-learning use; switch flipping tests precision, and drawer opening is common in simulated benchmarks.Source: B. Environments and Baselines · p. 4
• The main baselines are RAD and behavior cloning, and ablations remove demonstrations, contrastive pre-training, or data augmentation one at a time.Source: B. Environments and Baselines · p. 4

C. Results

• Across all six tested tasks, CoDER enabled a single robotic arm to learn optimal policies in 30 minutes on average (15–50 minutes; 20–80 training episodes), with final policies achieving 96% average success (86–100%).Source: C. Results · pp. 4–5
• CoDER learns all tasks, while RAD learns only one sparse-reward task without demonstrations and behavior cloning with 10 demonstrations succeeds only on simpler tasks.The simpler behavior-cloning successes are FetchReach and Light Switch; the figure compares normalized final rewards across simulated and real-robot tasks.Source: Figure 5 · p. 5
• The final policy achieves perfect success on Reach, Pickup, Light Switch, and Drawer Open, and high success on Move and Pull.Table I reports final-policy success rates over 30 episodes.Source: Table I · p. 5
• Ten demonstrations take 10 minutes of human operation to collect, while contrastive pre-training finishes within one minute on a single NVIDIA 2080Ti GPU.Source: C. Results · p. 5
• The same hyperparameters and camera setup are used across all six tasks.Source: C. Results · p. 5

Ablation setting

• The component ablations use simulated environments shown in Figure 7, including OpenAI Gym environments.The paper also compares pre-training and augmentation on real-robot tasks.Source: D. Ablations · p. 5, Figure 7 · p. 6

Demonstrations

• Ten demonstrations suffice for quick learning, whereas zero demonstrations lead to failure; one demonstration lets learning begin, but is less efficient.More demonstrations improve learning efficiency and reduce policy variance, while ten demonstrations are insufficient to learn an effective behavior-cloning policy.Source: D. Ablations · p. 5, Figure 6 · p. 6

Contrastive pre-training

• With 1600 contrastive pre-training iterations, the agent learns a successful policy, while the unpretrained runs fail to learn effectively.Without pre-training, the agent succeeds only once during an hour of training; on the real-robot move task, pre-training substantially boosts converged episode returns.Source: D. Ablations · p. 6, Figure 6 · p. 6

Online data augmentation

• Without online data augmentation, SAC fails to learn even dense-reward reaching, while augmented training achieves successful performance.The reaching reward is the negative Euclidean distance between the gripper and goal; this result motivates augmentation for sparse-reward tasks with less learning signal.Source: D. Ablations · p. 6, Figure 6 · p. 6

Imitation Learning

• Behavior cloning learns skills by regressing on demonstration data, while imitation-learning approaches generally require many demonstrations for each task, although recent advances can learn with a more modest amount [49],, [37],.Source: §V · p. 6

Reinforcement Learning

• Real-world robotic RL adoption is constrained by data inefficiency and safety; earlier efficient position-control methods still require hours and dense rewards, depth, or heavy computation, [52].Related RL methods using demonstrations address sparse-reward exploration with coordinate-state input rather than images, [54], [55].Source: §V · p. 6

Data Augmentation

• Data augmentation has supported unsupervised representation learning and pixel-based RL, with carefully implemented augmentation matching coordinate-state RL efficiency in prior work, [5], [22].It has also improved imitation-learning performance [16].Source: §V · pp. 6–7

Unsupervised Representation Learning

• Contrastive learning trains latent representations by bringing similar data points closer and pushing dissimilar points apart [31], [28],.Robotics work has used contrastive losses for viewpoint-invariant video representations and object representations,.Source: §V · p. 7

VI. CONCLUSION AND LIMITATIONS

• CoDER may need reset assistance and is not expected to generalize to qualitatively different unseen tasks or long-horizon sparse-reward problems.Performance is expected to degrade under substantial visual changes; the evaluated tasks have relatively short horizons.Source: §VI · p. 7