PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation

Zheyu Zhuang, Ruiyu Wang, Nils Ingelhag, Ville Kyrki, Danica Kragic

arXiv:2608.11870v1cs.RO

TL;DR

Vision-based BC needs robustness to visual domain shifts, while augmentations must preserve task-critical semantics and interactions. RoboSaGA applies saliency-guided, per-pixel superimposition without architectural changes, and experiments report improved robustness in simulated and real-world settings. Its main practical constraint is substantial saliency-computation overhead, especially for larger encoders.

  • Problem

    Conventional augmentations often fall short under lighting, shadow, distractor, and background shifts, while superimposition must preserve task-critical BC information.

  • Method

    RoboSaGA uses encoder-level saliency to adapt augmentation intensity per pixel, preserving task-critical regions while overlaying in-domain images onto OOD images.

  • Results

    RoboSaGA reduces the performance gap to 0.14 in simulations and 0.05 in real-world settings, outperforming Random Overlay.

  • Takeaways & Limitations

    Saliency-guided superimposition improves robustness to visual domain shifts across simulated and real-world BC experiments without requiring additional learnable modules or objectives.

  • Takeaways & Limitations

    Saliency computation remains costly: with a ResNet18 encoder on 84×84 inputs, it takes about 1.5 times longer than training, and larger encoders may require distillation.

Abstract

from arXiv · show

In vision-based behavior cloning (BC), conventional image augmentations such as Random Crop and Color Jitter often fall short under substantial visual domain shifts, including changes in shadows, distractors, and backgrounds. Superimposition-based augmentations, which blend in-domain and out-of-domain images, have shown promise for improving generalization in computer vision, but their suitability for BC remains uncertain because task-critical semantics, spatiotemporal relationships, and agent-target interactions must be preserved. To address this, we introduce RoboSaGA, a Saliency-Guided Augmentation method within the superimposition family tailored for vision-based BC. RoboSaGA dynamically adjusts augmentation intensity at the pixel level using policy-driven saliency, enabling aggressive augmentation in task-irrelevant regions while preserving task-critical information. It integrates seamlessly into existing architectures without requiring structural modifications or additional learning objectives. Experiments in both simulated and real-world settings show that RoboSaGA preserves in-domain performance while substantially improving robustness to visual domain shifts, including distractor and background changes, as well as lighting and shadow variations. Code is available at https://github.com/Zheyu-Zhuang/RoboSaGA.

1 Introduction

Vision-based behaviour cloning must generalize beyond task-related variation, but conventional augmentations struggle with visual domain shifts. RoboSaGA uses saliency-guided superimposition to preserve task-critical regions while improving robustness across lighting, shadows, distractors, and backgrounds.

  • BC datasets often prioritize task-related variability over visual diversity, making shifts in lighting, shadows, distractors, and backgrounds challenging.
  • RoboSaGA uses per-pixel saliency-derived blending to preserve task-critical regions while overlaying in-domain images onto OOD images.It requires no additional learnable modules or specific learning objectives and supports multi-view inputs and varied BC policies.
  • Simulated experiments show Random Crop and Colour Jitter struggle with distractors and background changes, while superimposition methods keep lighting and shadow gaps below 0.1.Colour Jitter’s shadow gap is 0.33.
  • The erase-based method handles distractors effectively but struggles with background variations.
  • Random Overlay reduces the performance gap from 0.60 to 0.24 in simulations and from 0.71 to 0.18 in real-world tests.These comparisons use Random Crop as the simulation baseline and the corresponding real-world baseline reported in the passage.
  • RoboSaGA further reduces the performance gap to 0.14 in simulations and 0.05 in real-world scenarios, consistently outperforming Random Overlay.

2 Related Work

BC policies integrate visual and proprioceptive observations into temporal embeddings, while augmentation methods combine geometric or colour randomization with superimposition. Superimposition varies through its blending masks and OOD inputs, but robotic manipulation requires preserving task-relevant content.

  • BC policies derive actions from temporal sequences of observation embeddings that integrate visual and proprioceptive inputs.Visual encoders process camera views, while a proprioceptive encoder handles state information.
  • Traditional augmentation randomizes colour space and image geometry through methods such as colour jittering, cropping, and rotation.
  • Superimposition augmentation forms an augmented image by combining in-domain and OOD images with a blending matrix.The blending matrix may be constant, randomly patched, or saliency-guided, while OOD inputs can include zeros, noise, or inter-class samples.
  • Robotic manipulation benefits from mild augmentation for in-domain generalization, but broader visual-domain coverage requires methods beyond Random Crop.Related approaches include language-conditioned masks and foundational modules that alter scene attributes.

3 Saliency-Guided Augmentation for Robotic Behaviour Cloning

RoboSaGA combines policy-encoder saliency with adaptive augmentation and computational strategies tailored to vision-based behaviour cloning. It selectively augments trajectories, updates saliency maps through a buffer, and preserves scalability across camera views and policy types.

  • RoboSaGA extracts saliency at the visual encoder level and uses adaptive per-pixel blending factors for vision-based manipulation.This design supports modularity and scalability across multiple camera views and policy types.
  • FullGrad computes saliency by back-propagating the full gradient to the full-resolution input while incorporating learned biases.The method reports sharper and more accurate saliency maps than approaches restricted to classification logits or specific layers.
  • Saliency clipping caps normalized scores at λ ∈(0, 1), enabling subtle overlays on task-critical regions to improve robustness against background changes.The normalized saliency is derived from FullGrad before clipping.
  • RoboSaGA selectively augments α of m trajectories to mitigate convergence risks from applying aggressive augmentation across an entire batch.Samples within trajectories are augmented independently.
  • Algorithm 1 initializes the buffer, processes visual encoders and batches, retrieves out-of-domain images, and manages saliency-map updates for augmentation.Its hyperparameters include α, β, and the saliency clipping factor λ.
  • A global saliency buffer stores past maps, updates the least recently updated β trajectories, and retrieves maps during augmentation to reduce FullGrad’s computational cost.Maps are stored as reduced-resolution 8-bit single-channel images; delayed updates showed no observable performance degradation.

4 Experiments

Experiments evaluate augmentation methods across simulated and real-world manipulation tasks, policies, and visual domain shifts. RoboSaGA combines saliency-guided overlay and erase mechanisms to reduce performance gaps, while results also expose policy- and task-dependent limitations.

  • Experimental Setup: Experiments cover simulated and real-world manipulation tasks using BC-MLP, BC-RNN, and Diffusion Policy under varied visual domain shifts.Simulation evaluates Lift, Square, Can, and Transport; real-world evaluation focuses on Toy.
  • Evaluation Protocol: The evaluation measures performance degradation as the gap between in-domain Random Crop performance and success under visual domain shifts.Smaller gaps indicate greater robustness, with metrics pooled across checkpoints, tasks, and policies.
  • Ablation: Guided Erase improves distractor resilience but is limited on background changes, whereas Random Overlay is more consistent but weaker against distractors.The ablation compares saliency-guided replacement with constant per-pixel blending.
  • Ablation: RoboSaGA reduces Random Overlay’s distractor gap from 0.31 to 0.06 and Guided Erase’s background gap from 0.55 to 0.13.These results support combining overlay and erase to address their complementary weaknesses.
  • Main Results: RoboSaGA further reduces Random Overlay’s gaps to 0.14 and 0.03 in simulation and to 0.05 in real-world tests.The real-world result averages performance across three policies on the Toy task.
  • Observations: Saliency does not always align with human-identified critical robot parts or targets across camera views and task stages.Examples show saliency shifting around the gripper, nut, peg, and hole during reaching, lifting, and insertion.

5 Limitations

The paper identifies substantial computational overhead for saliency extraction, weaker gains for some policies in Transport, and a narrow real-world evaluation scope. These limitations constrain efficiency, policy performance, and generality.

  • Computation Overhead: Saliency computation with a ResNet18 encoder on 84×84 inputs takes about 1.5 times longer than training.The authors suggest mid-feature approximations, alternative extractors, or knowledge distillation with offline saliency buffers as possible remedies.
  • Policy Performance in Transport: BC-MLP and BC-RNN show comparatively lower gains than Diffusion Policy under distractors and background changes in Transport.The authors associate this discrepancy with differences in extracting task-relevant information from images dominated by two manipulators.
  • Evaluation Scope: The real-world experiment evaluates a concentrated 3-DoF pick-and-place task, limiting conclusions beyond these specific conditions.The paper identifies broader task coverage as future work.

6 Conclusion

The paper concludes that superimposition-based augmentation improves robustness to visual domain shifts, while RoboSaGA further reduces performance gaps by adapting augmentation intensity with saliency. A saliency buffer preserves performance while reducing saliency-extraction time.

  • Conclusion: RoboSaGA reduces the performance gap to 0.14 in simulations and 0.05 in real-world settings, compared with Random Overlay gaps of 0.24 and 0.18.The reported relative improvements over Random Overlay are 41.6% and 72%, respectively.
  • Conclusion: RoboSaGA dynamically adjusts augmentation intensity using input saliency maps to improve robustness against visual domain shifts.The method targets background changes, object distractors, lighting, and shadows through superimposition-based augmentation.
  • Saliency Buffer: RoboSaGA with and without a saliency buffer achieves average success rates of 0.75 and 0.74 under visual domain shifts.No significant performance difference is observed between the two configurations.
  • Saliency Buffer: The saliency buffer reduces the computation time required for saliency extraction to one-third without enhancing performance.The comparison uses saliency updates for 10% of the batch and augmentation for 50% of the batch in the buffered setting.

Appendix B: Experimental Details

The appendix details the benchmark tasks, policy architectures, training settings, and out-of-domain image sources used to evaluate RoboSaGA. Experiments span four simulated manipulation tasks and augmentation data drawn from real and synthetic images.

  • Task Configuration: The task summary covers robot action, observation, proprioceptive, demonstration, rollout, horizon, and subtask specifications.These task metadata are organised in the appendix task-summary table.
  • Task Configuration: The experiments use four simulated RoboMimic tasks and one real-world pick-and-place task with a Franka Panda and 7-dimensional actions.The action space includes six end-effector pose degrees of freedom and gripper width.
  • Task Configuration: The simulated tasks include Lift, Can, Square, Transport, and Toy task descriptions covering grasping, placement, insertion, and coordinated manipulation.The appendix lists task-specific subtasks for these environments.
  • Policy Architectures: BC-MLP and BC-RNN process image observations with ResNet18 and spatial-softmax features before combining them with proprioceptive states.The resulting features feed an MLP or two-layer LSTM and then a Gaussian Mixture Model.
  • Policy Architectures: Diffusion Policy uses a hybrid-CNN with GroupNorm and horizons of 2, 8, and 16 for input, action, and action prediction.Its learning rate is set to 1 × 10^-4.
  • Augmentation Data: RoboSaGA uses approximately 6,000 out-of-domain images: 5,000 MSCOCO images and 1,000 synthetic pattern images.The images receive random rotations and brightness adjustments.

Lighting and Shadow Evaluation Setup

The lighting and shadow evaluation varies simulated light intensity and colour while comparing conditions with shadows enabled against training data in which shadows are disabled. The setup evaluates multiple policies across four simulated tasks.

  • Evaluation Setup: Lighting changes vary intensity and colour by adjusting the diffuse parameter between 0.2 and 0.8.The evaluation includes shadow rendering enabled while shadows are turned off in the training data.
  • Evaluation Setup: The simulated evaluation covers BC-MLP, BC-RNN, and diffusion policies across four simulated tasks.The setup examines lighting changes with shadow rendering both enabled and disabled.

Backgrounds and Distractors for Evaluation

The evaluation introduces visual domain shifts through lighting, shadows, distractors, and background changes in simulation and real-world settings. These shifts are instantiated with varied textures, materials, objects, and textile arrangements.

  • Simulation: Simulation evaluates lighting and distractor changes using varied table, floor, and wall textures plus common household objects.Textures include textiles, patterns, tiles, wooden flooring, and indoor or outdoor scenes; distractors include cans, bottles, cereal, bread, and lemons.
  • Real-world evaluation: Real-world evaluation varies textile combinations to shuffle backgrounds across both second-person and eye-in-hand camera views.The textile set is chosen to cover the fields of view of both camera types.
  • Real-world evaluation: Real-world distractors are arranged as clutter using objects drawn from a default distractor set.
  • Simulation: Some simulation distractors are removed when they duplicate task-specific objects, such as excluding the Coke can from the Can task.

Appendix C: Full Tables

The appendix provides success-rate tables for three policies under visual domain shifts and for real-world distractor and background variations. Reported table values are summarized as means with standard errors.

  • Reporting convention: All table data points are presented as mean ± standard error of the mean.
  • Simulation tables: Table 10 reports BC-MLP success rates for five augmentation methods under visual domain shifts.
  • Simulation tables: Table 11 reports BC-RNN success rates for five augmentation methods under visual domain shifts.
  • Simulation tables: Table 12 reports Diffusion Policy success rates for five augmentation methods under visual domain shifts.
  • Real-world table: Table 13 reports real-world success rates for each policy using Random Crop, Random Overlay, and RoboSaGA against distractor and background variations.

Appendix D: Interpretability Misalignment in Saliency Maps

The appendix compares policy saliency visualisations with human interpretation in the Transport task. BC-MLP appears more human-aligned, whereas history-dependent policies can be more robust to backgrounds but less interpretable.

  • Examples: Three representative Transport examples are used to examine potential misalignment between policy saliency and human interpretation.
  • Policy comparison: BC-MLP saliency maps align more closely with human intuition than those from BC-RNN and Diffusion Policy.
  • Camera-specific saliency: The left eye-in-hand camera focuses on the box handle, while the right eye-in-hand camera shows random focus without task-critical information.
  • Camera-specific saliency: Second-person views focus on the right arm or hand, while the left second-person camera omits the lid handle because the left eye-in-hand camera already supplies it.
  • Interpretability trade-off: History-dependent BC-RNN and Diffusion Policy achieve higher background robustness but produce saliency maps that are less interpretable to humans.
Loading 2608.11870v1…