PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive Red Teaming: Breaking Policies Without Breaking Robots

Anirudha Majumdar, Mohit Sharma, Dmitry Kalashnikov, Sumeet Singh, Pierre Sermanet, Vikas Sindhwani

arXiv:2502.06575v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Hardware evaluations can be costly when testing how robot policies perform under environmental changes. RoboART edits nominal observations to vary those factors and uses policy-specific anomaly detection to predict performance degradation. Across twelve factors, its average predicted-versus-real success-rate differences were 0.1 and 0.19 for two policies; targeted co-finetuning improved performance in predicted adverse conditions by 2–7x.

  • Problem

    Hardware evaluations to discover policy vulnerabilities across environmental changes can be prohibitively expensive, especially across diverse scenarios.

    Source: §1 · p. 2

  • Method

    RoboART edits nominal observations to vary environmental factors, then uses a policy-specific anomaly detector to predict performance degradation.

    Source: §1 · p. 2

  • Results

    Across twelve factors, average differences between predicted and real success rates were 0.1 and 0.19 for the two policies.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Co-finetuning with data from the three predicted most-adverse conditions boosted performance there by 2–7x and by 2–5x in conditions without collected data.

    Source: §1 · p. 3

    Predictions are not perfectly accurate, and current anomaly rates use only first-time-step observations from episodes.

    Source: §6.1 · p. 12

Abstract

from arXiv · show

Visuomotor policies trained via imitation learning are capable of performing challenging manipulation tasks, but are often extremely brittle to lighting, visual distractors, and object locations. These vulnerabilities can depend unpredictably on the specifics of training, and are challenging to expose without time-consuming and expensive hardware evaluations. We propose the problem of predictive red teaming: discovering vulnerabilities of a policy with respect to environmental factors, and predicting the corresponding performance degradation without hardware evaluations in off-nominal scenarios. In order to achieve this, we develop RoboART: an automated red teaming (ART) pipeline that (1) modifies nominal observations using generative image editing to vary different environmental factors, and (2) predicts performance under each variation using a policy-specific anomaly detector executed on edited observations. Experiments across 500+ hardware trials in twelve off-nominal conditions for visuomotor diffusion policies demonstrate that RoboART predicts performance degradation with high accuracy (less than 0.19 average difference between predicted and real success rates). We also demonstrate how predictive red teaming enables targeted data collection: fine-tuning with data collected under conditions predicted to be adverse boosts baseline performance by 2-7x.

1. Introduction

The paper addresses how to expose policy vulnerabilities to environmental changes without costly hardware evaluations in off-nominal conditions. It introduces RoboART, which edits nominal observations and uses anomaly detection to predict performance degradation.

1. Introduction

  • A pick-and-place diffusion policy’s success rate fell from ∼65% to ∼10% after a table-height change, while a nearby human left it roughly constant.The example illustrates how policy performance can defy intuitive predictions based on training variation.Source: §1 · p. 2
  • Predictive red teaming aims to expose vulnerabilities to environmental changes and predict their performance impact without hardware evaluations in those scenarios.Source: §1 · p. 2
  • RoboART edits nominal RGB observations to vary environmental factors, then uses a policy-specific anomaly detector to predict degradation.The detector compares policy-embedding distances against a conformal-prediction threshold.Source: §1 · p. 2
  • Across 500+ hardware experiments, average differences between predicted and real success rates were 0.1 and 0.19 across twelve factors for the two policies.RoboART also correctly ranked changed table height as more damaging than a human distractor.Source: §1 · pp. 2–3
  • Co-finetuning on the three predicted most adverse conditions raised performance 2–7x there and 2–5x in conditions without collected data.The latter result is reported as cross-domain generalization.Source: §1 · p. 3

2. Related Work

The related work spans automated red teaming, failure prediction, image editing, and off-policy evaluation. RoboART differs by forecasting a given policy’s performance under environmental changes using edited observations rather than runtime detection or simulation.

2. Related Work

  • Robotics red teaming remains nascent; prior work studies novel instructions [23] or jailbreaks [24], whereas this paper targets environmental changes without off-nominal hardware evaluations.Source: §2 · p. 3
  • Simulation can assess environmental generalization, but setting up an accurate RGB-policy simulator in a new environment may take months; RoboART instead uses policy training data and text descriptions.Source: §2 · p. 4
  • Prior failure-prediction and anomaly-detection methods operate at runtime; RoboART applies conformal-prediction-based detection to edited observations to forecast performance across conditions [42].Source: §2 · p. 4
  • Robotics image-editing work includes augmentation [53] [54] [55] [56] [57] and runtime visual generalization [60]; RoboART uses language-conditioned editing to create environmental variations [2].Source: §2 · p. 4
  • Off-policy evaluation estimates a policy using data from another policy [62] [63] [64], while predictive red teaming evaluates the same policy in nominal conditions to predict off-nominal performance.Source: §2 · p. 4

3. Problem: Predictive Red Teaming

Predictive red teaming uses nominal observations and policy access to assess vulnerabilities to selected environmental changes. Its goals are to rank those changes by degradation and predict performance under each one.

3. Problem: Predictive Red Teaming

  • The formalized problem is to expose policy vulnerabilities to environmental factors and predict their performance impact without hardware evaluations in those altered scenarios.Source: §3 · p. 4
  • A nominal scenario is a POMDP initialized in a particular state, and the nominal distribution captures ordinary environmental and task variations.The formulation assumes access to a nominal-observation dataset but not knowledge of the distribution itself.Source: §3 · p. 4
  • The red team receives a policy and nominal observations, including held-out nominal observations, and may query the policy on arbitrary observations.For the paper’s imitation-learning setting, nominal observations may come from the policy’s training data.Source: §3 · pp. 4–5
  • The red team chooses environmental factors and seeks to rank them by performance degradation and predict absolute performance under each factor.Ranking supports targeted data collection, while absolute estimates help characterize acceptable-performance conditions.Source: §3 · p. 5

4. RoboART: Predictive Red Teaming via Image Editing and Anomaly Detection

RoboART red-teams visuomotor policies by editing nominal observations to vary environmental factors, then predicting performance degradation with policy-specific anomaly detection. Its pipeline combines generative image editing with anomaly scoring of the edited observations.

4. RoboART: Predictive Red Teaming via Image Editing and Anomaly Detection

  • RoboART edits nominal observations to vary environmental factors, then predicts policy performance under each variation using anomaly detection.The approach targets visuomotor policies and factors such as background, lighting, and distractor objects.Source: §4 · p. 5

4.1. Generative Image Editing

  • Environmental factors can be selected at any granularity, according to deployment needs and plausible changes the robot may encounter.Examples include particular lighting conditions, distractor objects, and background colors.Source: §4.1 · p. 5
  • Nominal validation observations are edited with language instructions using Imagen 3, which supports realistic, globally consistent image edits.The model supports inpainting, outpainting, and colorization; examples include changing backgrounds, adding people, and changing lighting.Source: §4.1 · pp. 5–6
  • For multi-camera robots, each view is edited independently with the same prompt; Figure 3 shows original and edited overhead and wrist-camera observations.The example edit adds a novel object while maintaining per-view consistency in lighting, shadows, and composition.Source: §4.1 · p. 5, Figure 3 · p. 6
  • A vision-language model selects the best prompt-matching image from four edits or discards the observation if none reflects the desired change.Figure 4 depicts the VLM critic’s role in checking whether the edited image reflects the intended change.Source: §4.1 · p. 6, Figure 4 · p. 7

4.2. Predicting Performance via Anomaly Detection

  • For each environmental factor, RoboART uses the fraction of edited observations flagged as anomalous to estimate performance degradation.The anomaly rate is the proportion of edited observations exceeding the threshold; the method relates the nominal anomaly rate to nominal success rate when setting that threshold.Source: §4.2 · pp. 6–7
  • Policy-specific anomaly scores measure edited observations’ distance from nominal ones using the policy’s latent embeddings and cosine distance.The experiments use the mean of the k-nearest-neighbor cosine distances as the score.Source: §4.2 · p. 7
  • A conformal-prediction threshold calibrated on nominal validation observations bounds the rate of unseen nominal observations flagged as anomalous.The threshold uses an empirical quantile of validation-set anomaly scores.Source: §4.2 · pp. 7–8

5. Experiments

Across two visuomotor diffusion policies and twelve off-nominal factors, the experiments evaluate RoboART’s vulnerability and success-rate predictions, targeted data collection, and anomaly detection. Predictions correlate strongly with measured rankings, while fine-tuning on predicted-adverse conditions improves performance in both sampled and other conditions.

5. Experiments

  • Experiments comprise 500+ hardware trials across twelve environmental factors and two visuomotor diffusion policies with significantly different architectures.The factors include lighting and background colors, distractor objects, a nearby person, and changed table height.Source: §5 · pp. 8–9

5.1. How accurately does RoboART identify vulnerabilities and predict policy performance?

  • For the full-pipeline evaluation, RoboART uses 100 edited observations per factor and scores them by mean k-nearest-neighbor cosine distance in each policy’s embedding space.The nominal reference set is drawn from first-time-step observations in training episodes.Source: §5.1 · p. 9
  • RoboART identifies distinct vulnerability patterns: πhyb is most affected by table-height changes, while πdfn is more vulnerable to blue lighting and table-height changes.Figure 5 compares predicted and estimated rankings on the left and predicted and estimated success rates on the right.Source: §5.1 · p. 10, Figure 5 · p. 10
  • Under 0.19 average prediction error for both policies, with strong correlation between predicted and actual factor rankings.Table 1 quantitatively evaluates predicted versus real success rates; the reported error is roughly within the noise of estimating success rates from about 20 trials.Source: Table 1 · p. 10, §5.1 · p. 10
  • For πhyb, predictions remain accurate when the nominal reference-set size varies with a small k.Source: §5.1 · p. 11

5.2. How effective is RoboART in enabling policy improvement via targeted data collection?

  • 2–7x improvement in off-nominal conditions where data was collected, and 2–5x improvement in untrained background and lighting conditions.Additional demonstrations were collected for blue lighting, changed table height, and blue table background before co-fine-tuning.Source: §5.2 · p. 11, Figure 6 · p. 11

5.3. How accurately does anomaly detection predict performance degradation?

  • Anomaly-rate predictions from real observations show high Spearman correlation and low prediction errors for both policies.Table 2 evaluates success-rate predictions made from anomaly rates computed using real observations.Source: Table 2 · p. 12, §5.3 · p. 12
  • The anomaly-only predictions use about 20 real observations, five times fewer than the full pipeline’s 100 edited observations, making them more susceptible to noise.Source: §5.3 · p. 12

6. Conclusions

The paper presents RoboART as a baseline for predictive red teaming, using edited nominal observations and policy embeddings to identify vulnerabilities and predict performance degradation. It reports experiments across 500+ trials and describes limitations in edit realism, anomaly prediction, visual coverage, and factor selection.

6. Conclusions

  • Across 500+ trials, RoboART identified impactful environmental factors, predicted their performance impact, and supported policy improvement through targeted data collection.The conclusion describes RoboART as a baseline for predictive red teaming.Source: §6 · p. 12

6.1. Limitations and Future Work

  • Lighting edits do not change object shadows as real lighting does, and overhead and wrist edits can depict inconsistent geometry.The paper points to imperfect realism and multi-view consistency as edit-to-real gaps.Source: §6.1 · p. 12
  • Anomaly rates do not predict performance degradation perfectly, and the current approach uses only each episode’s first timestep.The authors suggest exploring multi-timestep sequences and other anomaly-detection methods.Source: §6.1 · p. 12
  • RoboART cannot identify vulnerabilities from environmental changes that are completely hidden from visual observations, such as changing object mass.Simulation-based red teaming is suggested for such cases, but requires bridging a potentially substantial sim-to-real gap.Source: §6.1 · p. 13
  • RoboART currently selects one set of environmental factors at the outset rather than iteratively expanding the search around predicted-poor conditions.The authors identify iterative exploration as a possible more sophisticated approach.Source: §6.1 · p. 13
  • The authors call for scalable methods to predict robot performance limits and hope RoboART encourages research on this underexplored direction.They frame RoboART as a baseline for further work.Source: §6 · p. 12, §6.1 · p. 13

B. Filtering Edits with a Vision-Language Model

The appendix describes how a vision-language model filters candidate edits for the desired environmental change. It uses shortened instructions because full prompts can make the model reject acceptable edits.

B. Filtering Edits with a Vision-Language Model

  • For each nominal observation, the pipeline generates four candidate edits and uses a vision-language model to select the best accurate edit or discard the observation.The VLM evaluates whether any candidate reflects the requested change; the experiments use Gemini Pro 1.5 [9].Source: B. Filtering Edits with a… · p. 21
  • The filtering prompt asks the VLM to reason about whether any of four images accurately reflects the shortened edit instruction and, if so, identify the best one.Source: B. Filtering Edits with a… · p. 21
  • Shortened edit instructions are used because full prompts can make the VLM overly critical and cause it to filter out acceptable edits.The filtering prompt asks the model to assess four images and choose the best match when one is acceptable.Source: B. Filtering Edits with a… · p. 21

C. Training and Policy Details

The policies are trained on 3K+ hardware demonstrations collected under fixed nominal conditions. The collection pipeline uses automated motion planning and annotation to produce grasping trajectories and object keypoints.

C. Training and Policy Details

  • The training set comprises 3K+ hardware demonstrations for grasping, collected with trajectory-optimization-based motion planners.Source: §C.1 · p. 21
  • The collection pipeline segments overhead-camera point clouds into objects, randomly selects objects for left-arm picks, and uses automated success detection to segment trajectories.Source: §C.1 · p. 21
  • Demonstrations are collected under fixed lighting, a fixed pink table background, and a fixed set of blocks, plush toys, small cans, and artificial fruits.Each episode also receives automatically annotated grasp-object keypoints as policy context.Source: §C.1 · p. 21

C.2. Hybrid Policy Architecture

The hybrid policy combines waypoint and trajectory action modes, while the diffusion policy uses only trajectory prediction. Both share vision and instruction encoders, and RoboART uses policy observation-instruction embeddings for anomaly detection.

C.2. Hybrid Policy Architecture

  • The hybrid policy has waypoint and trajectory action heads plus a mode-selection scalar, whereas the diffusion policy uses only the trajectory mode.The waypoint head predicts one SE(3) waypoint; the trajectory head predicts a sequence of joint angles.Source: §C.2 · p. 21, Figure 8 · p. 22
  • The hybrid policy executes waypoint actions with sequential quadratic programming and trajectory actions by interpolating predicted joint commands.Both heads are trained simultaneously; the mode selector is supervised to choose waypoint mode when the arm is far from an object and trajectory mode otherwise.Source: §C.2 · p. 22
  • Separate pretrained ViT encoders process overhead and wrist images, while a keypoint patch and skill embedding encode the grasp instruction.Proprioceptive features are encoded with an MLP.Source: §C.2 · p. 22
  • A transformer context fuser combines observation, instruction, and proprioceptive tokens; its observation-instruction embeddings support RoboART anomaly detection.In the hybrid policy, the waypoint-mode embedding feeds waypoint prediction and mode selection.Source: §C.2 · pp. 22–23

C.3. Diffusion Policy.

  • The diffusion policy shares the hybrid policy’s vision and multimodal instruction encoders but omits the waypoint/mode readout token.It directly predicts robot joint-angle trajectories using the trajectory diffusion head.Source: §C.3 · p. 23

C.4. Training and Inference Details

The policies use a batch size of 256, and an additional token learner reduces image observations to 128 tokens. Training and inference use a continuous-time diffusion model, with separate settings for targeted-data fine-tuning.

C.4. Training and Inference Details

  • Image observations are reduced to 128 tokens with a token learner, and policy training uses a batch size of 256.The token learner is added for high-dimensional image observations.Source: §C.4 · p. 23
  • The diffusion model uses a continuous-time implementation and a second-order Heun solver with 30 ODE steps at inference.The implementation is based on [77].Source: §C.4 · p. 23
  • An action horizon of 10 corresponds to one second of robot motion at the 10 Hz data-collection rate, and inference rolls out all 10 steps open-loop.Evaluation stops after a maximum of 30 policy steps.Source: §C.4 · p. 23
  • Targeted-data fine-tuning uses a 5e-6 learning rate, 4K-step linear warmup, and 20K total training steps.Source: §C.4 · p. 23

D.1. Ablations for RoboART

The ablation varies the nominal embedding set size and nearest-neighbor parameter for the anomaly detector, with effects that differ between the two policies.

D.1. Ablations for RoboART

  • Predictions for πhyb remain accurate with small k as |S_nom| varies, while πdfn benefits from smaller |S_nom| or larger k.Source: §D.1 · p. 24
  • RoboART scores an observation by the mean k-nearest-neighbor cosine distance to embeddings from a random subset of training data.The score measures dissimilarity from similar training observations in policy embedding space.Source: §D.1 · p. 24
  • The ablation tables compare predictions with empirical performance using Spearman rank correlation and average prediction error.Source: §D.1 · p. 24

E. Predicting Performance From Anomalies

Predictions from anomaly rates computed on real off-nominal observations correlate strongly with realized performance. These estimates use fewer observations than the full RoboART pipeline and are more susceptible to noise.

E. Predicting Performance From Anomalies

  • Predictions use approximately 20 real observations from each of the twelve off-nominal settings, five times fewer observations than the full pipeline and thus more susceptible to noise.Source: E. Predicting Performance From Anomalies · p. 25
  • For πhyb, conformal prediction sets the anomaly threshold to bound the nominal anomaly rate at 1 − R_nom^πhyb.Source: E. Predicting Performance From Anomalies · p. 25
  • Fig. 9 shows a strong correlation between predicted and realized performance using anomaly detection on real observations.The figure compares predicted rankings and success rates with their estimated true counterparts.Source: E. Predicting Performance From Anomalies · p. 25, Figure 9 · p. 26
  • For πdfn, the anomaly threshold used a higher nominal success-rate estimate (0.8 vs. 0.65), bounding the nominal-condition anomaly rate to 1 − 0.8 rather than 1 − 0.65.The adjustment may be due to the relatively small number of nominal observations (nval = 70) and the very high dimensionality (ℝ515×513) of πdfn’s embedding space.Source: E. Predicting Performance From Anomalies · p. 25
Loading 2502.06575v1…