PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning

Siyuan Huang, Yue Liao, Siyuan Feng, Shu Jiang, Si Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren

arXiv:2503.11646v1cs.RO

TL;DR

Real-scene imitation data collection is labor-intensive, while synthetic data can face a domain gap and environmental changes require costly human resampling. ADC pairs teleoperation with an adversarial operator who dynamically changes visual conditions and linguistic task goals during data collection. ADC-trained policies show significant robustness to dynamic visual perturbations, achieving a 0.88 success rate when spatial positions change.

Abstract

from arXiv · show

The pursuit of data efficiency, where quality outweighs quantity, has emerged as a cornerstone in robotic manipulation, especially given the high costs associated with real-world data collection. We propose that maximizing the informational density of individual demonstrations can dramatically reduce reliance on large-scale datasets while improving task performance. To this end, we introduce Adversarial Data Collection, a Human-in-the-Loop (HiL) framework that redefines robotic data acquisition through real-time, bidirectional human-environment interactions. Unlike conventional pipelines that passively record static demonstrations, ADC adopts a collaborative perturbation paradigm: during a single episode, an adversarial operator dynamically alters object states, environmental conditions, and linguistic commands, while the tele-operator adaptively adjusts actions to overcome these evolving challenges. This process compresses diverse failure-recovery behaviors, compositional task variations, and environmental perturbations into minimal demonstrations. Our experiments demonstrate that ADC-trained models achieve superior compositional generalization to unseen task instructions, enhanced robustness to perceptual perturbations, and emergent error recovery capabilities. Strikingly, models trained with merely 20% of the demonstration volume collected through ADC significantly outperform traditional approaches using full datasets. These advances bridge the gap between data-centric learning paradigms and practical robotic deployment, demonstrating that strategic data acquisition, not merely post-hoc processing, is critical for scalable, real-world robot learning. Additionally, we are curating a large-scale ADC-Robotics dataset comprising real-world manipulation tasks with adversarial perturbations. This benchmark will be open-sourced to facilitate advancements in robotic imitation learning.

I. INTRODUCTION

Real-scene imitation data is costly to collect and difficult to scale across diverse conditions. ADC uses real-time human perturbations during a demonstration to increase its diversity, and the paper reports stronger generalization and robustness with fewer demonstrations than conventional collection.

I. INTRODUCTION

  • Real-scene teleoperation is labor-intensive and costly, while synthetic data can face a domain gap and environmental changes require costly human resampling.Conventional demonstrations also contain redundant frames, repeated instructions, and correlated actions.Source: §I · pp. 1–2
  • ADC couples a tele-operator with an adversarial operator who changes scene configurations and task instructions during one episode, prompting adaptive action replanning.Perturbations include repositioning or rotating objects, changing backgrounds, and paraphrasing instructions.Source: §I · p. 2
  • ADC compresses recovery behaviors, compositional instruction variation, and environmental changes into unified demonstrations through real-time operator adaptation.The approach is presented as a way to increase demonstration efficiency and model robustness to perceptual, linguistic, and physical uncertainty.Source: §I · p. 2
  • The paper reports superior generalization and robustness from ADC-trained models with significantly fewer demonstrations than larger conventional datasets.The evaluation dataset is described as containing 200 demonstrations, including ADC demonstrations totaling 96K frames and conventional demonstrations totaling 90K frames.Source: §I · p. 2

II. RELATED WORK

Prior data-collection work emphasizes dataset scale and diversity, while other approaches explore efficient collection and vision-based imitation learning. This paper positions ADC as extending collection strategies to language variation and real-time human intervention.

A. Data Collection in Robot Learning

  • Data-collection studies often scale datasets for diversity, while efficiency-focused work has largely examined visual compositional generalization rather than language variation.DROID is cited as containing 76k trajectories across more than 500 scenes.Source: A. Data Collection in Robot Learning · p. 2
  • ADC is framed as extending data collection across visual and language dimensions and dynamically perturbing actions rather than only resetting initial conditions.The paper connects this intervention to human-robot interaction during deployment.Source: A. Data Collection in Robot Learning · p. 2

B. Vision-based Imitation Learning

  • Vision-based imitation combines image observations with language goals, and scaling these models increases the need for diverse, high-quality robot data.Prior work also explores diffusion policies and large-scale unlabeled action videos.Source: B. Vision-based Imitation Learning · p. 2

C. Generalization and Robustness in Robotic Policy

Robotic policies must generalize to unseen objects and environments while remaining robust to human intervention. The paper situates ADC within VLA training, whose conventional fixed episodes contain redundant and fragmented training units.

C. Generalization and Robustness in Robotic Policy

  • Generalization to unseen objects and environments must be paired with robustness to human interventions in dynamic human-robot interactions.Prior compositional-generalization work is described as primarily studying small-scale policies.Source: C. Generalization and Robustness in… · p. 2

III. APPROACH

  • The approach is organized around VLA data structure, ADC collection, policy integration, and VLA training.The overview describes dynamic visual perturbations and adaptive linguistic challenges as increasing information density and state-space coverage.Source: §III · p. 3, Figure 2 · p. 3

1) VLA Architecture and Training Data Structure:

  • Memory-less VLA models use multi-view observations and language instructions to predict actions, while conventional datasets group training units into fixed episodes.The paper identifies over 70% of consecutive units as sharing redundant visual-language contexts and notes that equivalent units remain isolated across episodes.Source: 1) VLA Architecture and Training Data… · p. 3

2) Density Optimization:

Density optimization shifts data quality from the number of episodes toward diversity within each demonstration. ADC seeks that diversity through mid-episode visual and instruction perturbations and their combinations.

2) Density Optimization:

  • The paper reframes data quality around information density per demonstration rather than the number of episodes.Its perturbation strategy enriches unit diversity within individual demonstrations.Source: 2) Density Optimization · p. 3
  • ADC perturbs object poses, lighting, and instructions mid-episode, including synergistic visual-linguistic combinations.Functionally equivalent units may differ in spatial positions or prompt diversity under task constraints.Source: 2) Density Optimization · p. 3
  • Adaptive responses to dynamic perturbations compress hundreds of traditional trials into single demonstrations while increasing unique-unit coverage and reducing temporal redundancy.The resulting dataset structure is described as aligning with VLA’s Markovian requirements.Source: 2) Density Optimization · p. 3

B. ADC: Adversarial HiL Framework

ADC turns teleoperation into a two-operator process: one operator performs the task while the other introduces real-time perturbations. Its visual perturbations vary object positions and grasp conditions, requiring adaptive repositioning.

B. ADC: Adversarial HiL Framework

  • The tele-operator executes tasks while an adversarial operator dynamically introduces controlled visual and linguistic perturbations during collection.The perturbations aim to increase per-demonstration diversity while preserving physical plausibility.Source: B. ADC: Adversarial HiL Framework · p. 3
  • The hardware setup uses Aloha for conventional-policy experiments with visual distractors and AgiBot G1 for VLA experiments with dynamic perturbations.Figure 3 depicts the hardware used for both collection and evaluation.Source: Figure 3 · p. 4
  • Target objects are initialized from task-specific Gaussian position distributions and may be moved within the worktable when the end-effector misses the grasp-proximity threshold.The passage does not specify the threshold for this positional reset.Source: B. ADC: Adversarial HiL Framework · p. 4
  • When the end-effector comes within 15 cm, perturbation impulses alter object position and orientation, requiring a new grasp configuration.The same mechanisms also apply to target containers.Source: B. ADC: Adversarial HiL Framework · p. 4

1) Visual Perturbations:

ADC perturbs both objects and instructions, prompting the tele-operator to adapt within an episode. The paper demonstrates mid-execution instruction changes, while noting that dynamic spatial-language understanding is a separate focus.

1) Visual Perturbations:

2) Linguistic Perturbations:

  • Mid-execution instructions can change the target object, the required action, or both, requiring immediate adaptation while maintaining task stability.Source: 2) Linguistic Perturbations · p. 4
  • Spatial descriptors such as “left” or “near” can be redefined as object positions change, but the paper primarily demonstrates the first mechanism because spatial understanding is a separate VLA research focus.Source: 2) Linguistic Perturbations · p. 4
  • Real-time adaptation compresses many static variations into unified episodes, with trajectory replanning, grasp recovery, and compositional task switching producing diverse data units.Source: 2) Linguistic Perturbations · p. 4

C. ADC in Conventional Robotic Policy

The paper evaluates ADC in conventional robotic policies and VLA fine-tuning, using visual and linguistic perturbations to broaden the challenges represented in demonstrations. The VLA setup excludes kiwi placement from training for unseen-task evaluation.

C. ADC in Conventional Robotic Policy

  • With equal volumes of adversarial and conventional demonstrations, the conventional-policy experiment uses Aloha and ACT for a plastic-cup-to-plate task involving grasping a transparent handle.The adversarial collection varies spatial positions and dynamically perturbs the target.Source: C. ADC in Conventional Robotic Policy · p. 4
  • The conventional-policy results are qualitative: ADC-trained policies show improved robustness and generalization, but exhibit oscillatory grasping when placement height varies.The authors attribute the oscillation to depth-estimation difficulty under greater state variance and smaller-model capacity limits.Source: C. ADC in Conventional Robotic Policy · p. 4

D. ADC in VLA

  • For VLA experiments, ADC is applied to π0 on AgiBot G1, with object displacement before grasping and container repositioning after preplacement.The collection-strategy comparison is summarized in Table I.Source: D. ADC in VLA · p. 4, Table I · p. 5
  • VLA instructions can change after grasping; task-decomposed grasping and placement labels maintain temporal continuity, while kiwi placement is excluded from training as an unseen evaluation task.The example changes the destination plate after the orange has been grasped.Source: D. ADC in VLA · pp. 4–5

IV. RESULTS AND ANALYSIS

In static evaluation, ADC-trained VLA policies outperform traditional-data baselines across tested heights and positions, including compositional generalization. At the most challenging table height, the reported average success rates are 0.72 and 0.13, respectively.

IV. RESULTS AND ANALYSIS

  • The VLA evaluation fine-tunes π0, with kiwi placement withheld from training to test generalization to an unseen task; each task is run 10 times.The reported metric is success rate.Source: §IV · p. 5

A. Evaluating ADC in Static Environments

  • The traditional-data baseline falls to 0.0 for all tasks under varied positions, while ADC demonstrates strong compositional generalization on Place-C.Normal positions place the target near the table center; varied positions sample across the full work area.Source: A. Evaluating ADC in Static Environments · p. 5
  • 0.72 versus 0.13 average success rate at the most challenging height (Var. 3) for ADC-trained versus traditional-data VLA policies; ADC outperforms across all tested heights and positions.Table II reports static-environment results and highlights Place-C for compositional-generalization evaluation.Source: Table II · p. 5, A. Evaluating ADC in Static Environments · p. 5

B. Evaluating ADC in Dynamic Environments

ADC is evaluated under dynamic visual and linguistic changes and masked-camera failures. It remains robust to visual perturbations and camera masking, while instruction changes during grasping are more challenging.

B. Evaluating ADC in Dynamic Environments

  • ADC achieves a 0.88 success rate under dynamic visual changes, comparable to static evaluation, while the baseline fails entirely.Instruction changes notably affect performance during the “During Grasp” phase, when the policy must halt, reassess, and choose a new target.Source: Table III · p. 5, B. Evaluating ADC in Dynamic… · p. 5

C. Evaluating ADC in Sensor-Failure Scenarios

  • With a target camera masked, ADC maintains robustness across masking settings and shifts attention toward functional cameras rather than irrelevant features.The masking experiment replaces the target-camera input with an all-zero matrix.Source: Table V · p. 6, C. Evaluating ADC in Sensor-Failure… · p. 6

D. Further Analysis

Further analysis compares data efficiency and examines visual coverage, dynamic human-robot interaction, scene generalization, and failure recovery. ADC’s diverse demonstrations support robustness across these settings, with a reported trade-off between collection effort and data efficiency.

Data Efficiency

  • With only 20% ADC data, the model demonstrates significantly greater robustness and positional generalization in static and dynamic environments than a model trained with 100% traditionally collected data.The evaluation used visual and linguistic perturbations on standard table height (Var.1).Source: D. Further Analysis · p. 6

Complete Object Observation Coverage

  • ADC provides wider orange viewpoints and more focused attention on functional cameras under masking than traditional collection.The paper links broader visual observation coverage to improved generalization and reports more concentrated attention under camera masking.Source: D. Further Analysis · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

Dynamic HRI

  • In human-robot interaction scenarios, ADC-trained models adapt when a person holds and moves the target during grasping.The evaluation also considers language perturbation; the authors describe dynamic adaptation as a prerequisite for future HRI.Source: Figure 6 · p. 7, D. Further Analysis · p. 7

Shuffle Buffer

  • ADC trajectories contain diverse motion and semantic information within a single trajectory, reducing engineering effort for shuffle-buffer management.The paper contrasts this with interleaving images from different trajectories to diversify training batches.Source: D. Further Analysis · p. 7

Failure Recovery Analysis

  • ADC-trained models can retry after an empty grasp, using a secondary grasp attempt after detecting an empty-gripper state.The authors attribute this capability to varied grasp initialization poses and tele-operator retries following perturbation-induced failures.Source: Figure 7 · p. 7, D. Further Analysis · p. 7

Data Collection Cost Discussion

  • ADC’s episodic time cost is less than 2× conventional collection, alongside a reported greater than 5× improvement in data efficiency.ADC uses two operators, but the adversarial operator requires less expertise and participates intermittently.Source: D. Further Analysis · p. 7

Novel Scene Generalization

  • Despite collection on a white table surface, ADC-trained policies generalize to unseen scenes with tablecloths, unlike the traditional-data model.The authors attribute this to both pretrained visual encoders and ADC’s visually adversarial images.Source: D. Further Analysis · p. 7

V. CONCLUSION

The conclusion presents strategic adversity during data acquisition as a route to more efficient robotic learning. It argues that human-driven perturbations can pack varied responses into demonstrations and that acquisition quality deserves emphasis alongside quantity.

Conclusion

  • The paper presents strategic adversity during data acquisition as fundamental to efficient robotic learning.Its ADC framework uses human-driven visual and linguistic perturbations during acquisition.Source: §V · p. 8
  • ADC is described as compressing hundreds of static variations into single demonstrations through real-time human adaptation and physics-constrained perturbations.The conclusion says this addresses data redundancy and domain gaps in conventional pipelines.Source: §V · p. 8
  • The authors argue that purposeful perturbation during acquisition, rather than dataset quantity alone, may support generalizable embodied intelligence.The conclusion calls for re-examining robotic data ecosystems around data quality.Source: §V · p. 8
Loading 2503.11646v1…