PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning

Yusuf Umut Ciftci, Joonwon Kang, Ajay Suresha Sathya, Somil Bansal

arXiv:2610.17709v1cs.LGcs.AIcs.RO

TL;DR

Behavior cloning treats demonstration states uniformly, although small action errors can be inconsequential in some states and lead to failure in others. The paper introduces SCRIM, which estimates state criticality through adversarial bounded disturbances around an imitation policy. SCRIM-guided reweighting and data selection improve imitation, and selected subsets using only a quarter of the data match or exceed uniform BC trained on the full dataset.

  • Problem

    Behavior cloning weights demonstration states equally even though small action errors can be recoverable in some states and cause task failure in others.

    Source: §1 · p. 2

  • Method

    SCRIM estimates state criticality as worst-case outcome degradation from bounded action disturbances, using an adversarial policy and critic around a fixed imitation policy.

    Source: §4 · p. 4

  • Results

    With 25% of the dataset, SCRIM-selected subsets achieve the best average success rate, win on three of four tasks, and match or exceed uniform BC trained on the full dataset.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    SCRIM identifies semantically meaningful critical phases and supports both loss reweighting and data curation for stronger behavior-cloning policies.

    Source: §1 · p. 2

    SCRIM still requires environment interaction, may be difficult where real-world failures are costly, and cannot discover strategies or recovery behaviors absent from the demonstrations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Imitation learning typically treats every demonstration state as equally important, optimizing a behavior-cloning objective uniformly across the dataset. Yet not all states are created equal: in a small number of critical states, even a slight deviation from the expert action can lead to a failure, while in many other states such errors are recoverable and inconsequential. We formalize state criticality in imitation learning as the worst-case degradation in downstream value induced by a bounded perturbation of the expert action at a given state. We then propose SCRIM, a practical State CRIticality Measure that estimates this quantity by training an adversarial disturbance policy and a value critic around a fixed imitation policy. SCRIM assigns each demonstration state a score reflecting how vulnerable the task outcome is to small, policy-relevant action errors. We show that these scores can be used to reweight behavior-cloning losses, improving imitation at consequential states and increasing overall task performance. We further show that SCRIM enables data curation by selecting compact, high-value subsets of demonstrations that preserve performance while substantially reducing the amount of training data. We evaluate SCRIM across various simulated and real-world robotic manipulation tasks, and demonstrate that SCRIM improves policy training, enables efficient data selection, and provides interpretable insights into which moments in a demonstration are most critical for task success. Project website: https://sia-lab-git.github.io/SCRIM/

1 Introduction

Behavior cloning treats demonstration states uniformly, although small action errors can be recoverable in some states and derail a task in others. The paper formalizes this difference as state criticality and proposes SCRIM to guide policy training and demonstration selection.

1 Introduction

  • Behavior cloning can be brittle outside demonstration coverage, while collecting more demonstrations and manually demonstrating recovery from critical moments are costly.Residual reinforcement learning is another option, but sparse-reward manipulation can require prohibitively many environment interactions or carefully engineered simulation.Source: §1 · pp. 1–2
  • A small set of states can determine task success: errors during grasp or placement alignment may derail cube stacking, while similar errors during earlier reaching can be recoverable.Criticality scores could prioritize consequential states during BC training and identify compact demonstration subsets that match or exceed full-dataset performance while reducing training and storage costs.Source: §1 · p. 2
  • SCRIM estimates how much bounded, policy-relevant action errors can degrade task outcomes and identifies consequential states that geometric and value-based baselines miss.The authors report that SCRIM-guided loss reweighting and data selection significantly improve behavior cloning over standard training and alternative criticality measures.Source: §1 · p. 2
  • The paper introduces a formal state-criticality notion, a practical estimator, and demonstrations that it can identify meaningful states and improve imitation through reweighting and data curation.The evaluation spans simulated and real-robot manipulation demonstrations.Source: §1 · p. 2

2 Related Work

Prior work studies demonstration quality, dataset curation, and robustness to deviations from demonstrations. This paper uses one state-criticality score both to curate data and to emphasize states in the imitation objective.

2 Related Work

  • Prior data-curation approaches select or reweight demonstrations using task distributions, surrogate models, quality estimates, or data-collection strategies.Dataset quality and composition shape downstream performance.Source: §2 · p. 3
  • This work formalizes the importance of each state with a criticality score used for both demonstration curation and imitation-loss reweighting.The score targets states where small mistakes matter most.Source: §2 · p. 3
  • Residual reinforcement learning trains corrective policies atop fixed controllers, but sparse-reward manipulation can make such corrections interaction-intensive.Rather than correcting the base policy, this paper trains an adversarial residual to degrade it, motivated by the authors’ claim that failure-inducing actions are easier to explore than corrective ones.Source: §2 · p. 3

3 Background and Problem Formulation

The section frames imitation learning as supervised fitting of expert actions and highlights that action errors have unequal consequences across demonstration states. Its objective is to identify states where precise action selection matters most.

3 Background and Problem Formulation

  • A Markov decision process specifies states, actions, transitions, rewards, an initial-state distribution, and a discount factor.A policy maps states to action distributions, or to actions when deterministic.Source: §3 · p. 3
  • Behavior cloning fits a parameterized policy to expert actions as a supervised-learning problem, weighting all demonstration states equally.That uniform objective does not reflect how consequential errors differ across states.Source: §3 · p. 3
  • The paper uses the Q-function to assess action sensitivity and seeks a state-dependent measure that can diagnose policy weaknesses or prioritize improvement.The measure may also guide data collection or targeted intervention during deployment.Source: §3 · pp. 3–4
  • The same action error may be recoverable in one state but cause task failure in another, so criticality concerns sensitivity of task success to the selected action.Non-critical states admit a wider range of actions from which the policy can still recover.Source: §3 · p. 3

4 Method

SCRIM estimates state criticality by measuring the worst outcome degradation caused by a bounded perturbation around an imitation policy. An adversarial disturbance and critic are trained in a residual MDP, then scores on demonstrations are evaluated relative to expert actions.

4 Method

  • The method freezes the imitation policy and treats a bounded residual disturbance as the sole decision variable in an induced residual MDP.The disturbance budget models small action errors; the state space, initial-state distribution, and discount factor are inherited from the original MDP.Source: §4 · p. 4
  • The residual MDP uses a sparse outcome reward of −1 for success and 0 otherwise, so larger critic values correspond to worse outcomes.The critic evaluates the combined action executed in the environment.Source: §4 · p. 4
  • SCRIM defines criticality as the worst-case degradation in outcome value from a small admissible action disturbance, relative to faithfully following the expert.Large scores indicate that an in-budget error can substantially worsen outcomes; scores near zero indicate forgiving states.Source: §4 · p. 4
  • Although the adversary is trained around the learned imitation policy, demonstration-state criticality is evaluated around the expert action.This tests whether a small prediction error around the demonstrated action can increase downstream failure risk.Source: §4 · pp. 4–5
  • A disturbance policy is trained jointly with the critic to seek the most failure-prone admissible disturbance, using soft actor-critic.The disturbance is bounded per action coordinate, and the conservative scoring critic is the minimum of a clipped double-Q pair.Source: §4 · p. 5
  • Before either downstream use, SCRIM standardizes scores and maps them to a temperature-softmax sampling distribution calibrated to a target effective sample size.The calibrated scores can then upweight critical samples for policy improvement.Source: §4 · p. 5

5 Experiments

The experiments test SCRIM for identifying critical states, improving behavior cloning, curating demonstrations, and transferring to hardware. SCRIM-weighted training and selection improve or preserve performance across simulation and hardware, while the state-level analyses show which moments are consequential.

5 Experiments

  • The evaluation spans four simulated manipulation tasks and two Franka Panda hardware tasks, using SAC adversarial disturbances around fixed policies; hardware adversarial training used 50 StackCube and 150 TapeHang rollouts.Simulation includes Hammer, StackCube, PegInsertion, and TurnFaucet; hardware policies finetune π0.5.Source: §5 · p. 5, §5.1 · pp. 5–6, §5.2 · p. 8

5.1 Simulation Results

  • On Hammer, SCRIM identifies pre-grasp alignment over the handle as critical, while the baselines peak elsewhere; faucet-model mean SCRIM ranks broad, forgiving handles easier than models requiring precise grasping and turning.The Hammer measures are min–max normalized to a common scale.Source: §5.1 · pp. 6–7, Figure 2 · p. 6, Figure 3 · p. 6
  • SCRIM weighting improves average success over uniform BC, gaining on three of four tasks and essentially matching it on PegInsertion while outperforming other baseline weights.The strongest baseline, Q-function variance, lands barely above uniform BC.Source: Table 1 · p. 7, §5.1 · p. 7
  • For StackCube, action error is concentrated at higher-criticality states under uniform BC, and SCRIM weighting flattens that relationship while raising success; improvement appears limited when critical-state errors are already low.The authors attribute this boundary to SCRIM’s lack of RL-style exploration for new behaviors.Source: §5.1 · p. 7, Figure 4 · p. 7
  • Under matched interaction budgets, residual RL falls below the base policy on all three ManiSkill tasks, whereas SCRIM-weighted BC improves StackCube and TurnFaucet; Hammer is an exception, and SCRIM reaches residual RL’s peak within 500k interactions.The comparison indicates that SCRIM’s interaction efficiency is task-dependent and the approaches are complementary.Source: Table 2 · p. 8, §5.1 · p. 8
  • With 25% of the demonstrations, SCRIM-selected subsets match or exceed full-data uniform BC and attain the best average success, winning on three of four tasks.Uniform selection is narrowly ahead on PegInsertion.Source: Table 3 · p. 8, §5.1 · p. 8

5.2 Hardware

  • On hardware, SCRIM-weighted training raises StackCube success from 56% to 76% and TapeHang from 56% to 88%, while 25% SCRIM-selected subsets outperform uniform selection on both tasks.These results use a finetuned π0.5 policy and 25 evaluation rollouts per task-policy pair.Source: Table 4 · p. 9, §5.2 · p. 9

6 Limitations and Future Work

SCRIM still requires environment interaction, and its improvements remain bounded by the demonstrations’ support; future work could reduce interaction needs or use criticality to guide data collection.

6 Limitations and Future Work

  • SCRIM estimates criticality without expert supervision but still requires environment interaction, which may be difficult when real-world failures are costly.The authors suggest learning criticality in simulation, reconstructing real demonstrations in simulation, or using logged failure data with offline RL.Source: §6 · p. 9
  • SCRIM can emphasize consequential demonstrated states but cannot discover strategies or recovery behaviors missing from the dataset.The authors propose using criticality to guide exploration or targeted data collection near high-criticality states.Source: §6 · p. 9
  • The authors identify language-conditioned critics that generalize across objects, tasks, and embodiments as a future direction.Faucet and hardware experiments suggest this may be feasible because SCRIM can train across task variations with task instructions in the observation.Source: §6 · p. 9

A.1 SCRIM algorithm

SCRIM has two phases: it trains an adversarial disturbance policy and value critic around a frozen imitation policy, then uses the converged components to score demonstration states.

A.1 SCRIM algorithm

  • Phase 1 trains an adversarial disturbance policy and value critic in the residual MDP using SAC.The base controller is the frozen imitation policy; the algorithm’s particular base-policy architecture and action-chunking details are abstracted away.Source: §A.1 · p. 13
  • Phase 2 applies the criticality estimator ∆Q(s) to each demonstration state using the conservative critic and deterministic adversary.The resulting per-state scores feed the sampling-distribution calibration pipeline.Source: §A.1 · p. 13
  • After convergence, SCRIM forms a conservative critic from the twin critics and a deterministic adversary from the disturbance policy.The deterministic disturbance is bounded coordinate-wise by ϵ.Source: §A.1 · p. 13

A.2 Simulation setup and per-environment details

The simulation study covers four manipulation tasks across MetaWorld and ManiSkill2, using task-specific observations, base policies, and demonstration datasets.

A.2 Simulation setup and per-environment details

  • The setup includes MetaWorld Hammer and ManiSkill2 Stack Cube, Turn Faucet, and PegInsertion, with actions normalized to [−1, 1]^da.Hammer uses 84×84 RGB images plus four proprioceptive values; the ManiSkill2 tasks use state observations.Source: §A.2 · p. 13
  • Hammer uses a deterministic CNN+MLP trained on 20 expert demonstrations, while ManiSkill2 uses a Behavior Transformer with 20 time-steps of history.StackCube, TurnFaucet, and PegInsertion use 974, 1008, and 986 demonstrations, respectively; TurnFaucet spans 10 faucet models.Source: §A.2 · p. 13
  • For Hammer, the SAC adversary trains for 5×10^5 steps with a disturbance bound of 0.20 per axis.The disturbance bounds and interaction budgets for each task are listed in Table 5.Source: §A.2 · p. 13

A.3 Sampling-distribution calibration

The calibration procedure standardizes criticality scores, maps them to a temperature-controlled softmax distribution, and selects temperature by targeting fractional effective sample size; tested targets keep weighted BC above uniform BC.

A.3 Sampling-distribution calibration

  • Each criticality score is z-score standardized using a small ξ > 0 for numerical stability, then mapped to a temperature-T distribution.The softmax distribution is proportional to exp(z_i/T); standardization sets a common temperature scale across measures with different raw-score magnitudes.Source: §A.3 · pp. 13–14
  • The procedure chooses temperature by bisection to reach a target fractional effective sample size, ESS% = τ.ESS% is 100% for a uniform distribution and decreases as probability mass concentrates on fewer transitions; τ is a hyperparameter.Source: §A.3 · p. 14
  • Across tested ESS% targets, SCRIM-weighted BC remains above uniform BC throughout.Table 6 reports the selected targets, resulting temperatures, and success rates across targets.Source: §A.3 · p. 14, Table 6 · p. 14

A.4 Baseline details

The appendix compares SCRIM with sampling, geometry, action-magnitude, and value-based baselines, then examines hardware computation, simulation-based scoring, perturbation bounds, score-gated attacks, and critic error. Across these tests, high-SCRIM states identify vulnerability to specific failure-inducing disturbances, while score reliability depends on resolving local value gaps.

A.4 Baseline details

  • The baselines include uniform state sampling, end-effector distance to the target, negative expert-action magnitude, and BC-policy Q-function variance or Q-value discrepancy.Distance uses scene geometry and requires target-position information; action magnitude treats smaller actions as more critical, while high Q variance and large Q discrepancy indicate greater sensitivity to action choice.Source: §A.4 · pp. 14–15
  • Per-method criticality is plotted along StackCube, TurnFaucet, and PegInsertionSide demonstrations, with measures min–max normalized to a common scale.The action-error comparison plots BC action-error percentiles against SCRIM percentiles under uniform and SCRIM-weighted BC.Source: Figure 6 · p. 15, Figure 7 · p. 15, Figure 8 · p. 16

A.5 Hardware setup and adversary details

  • SCRIM’s real-robot score is the failure-value increase around the demonstrated action in chunked action space, with each chunk’s value held constant over its control ticks.The hardware setup uses a residual adversary on the executed four-step action chunk, scaled by task-specific residual bounds and clipped to robot safety limits.Source: §A.5 · p. 16

A.6 Learning SCRIM without physical interaction via real2sim

  • 40% to 66%: SCRIM-weighted training raises simulation success using scores learned entirely in simulation from reconstructed hardware demonstrations.Simulation-derived scores correlate with hardware-interaction scores and recover critical phases around grasping and placement, despite the real2sim gap; transfer back to the physical robot remains unverified.Source: §A.6 · p. 17

A.7 Choosing the perturbation bound

  • SCRIM remains robust over a broad range of perturbation bounds at a fixed effective-sample-size target.The bound is calibrated so random residuals neither leave success unchanged nor collapse it, conditions that would respectively make disturbances ineffectual or render every state critical.Source: §A.7 · p. 17, Table 8 · p. 17

A.8 Causal validation of SCRIM

  • At high-SCRIM states, adversarial disturbances reduce success at least twice as much as at the same fraction of random states, while uniform noise barely changes success.The test gates disturbances to selected steps and compares trained-adversary and uniform-noise perturbations against random-step selection.Source: Table 9 · p. 18, §A.8 · p. 18

A.9 Robustness to critic error

  • At critic-noise scale c = 0.003 (SNR 4.1–18.4), success-rate drops remain comparable to noise-free values; at c = 0.01, they remain 8–19 points.When noise approaches the local ΔQ scale (SNR ≤1), effects become task-dependent; at c = 1, drops collapse to −1 to 5 points. Halfway-trained critics also vary by task, so general robustness to critic undertraining is not claimed.Source: Table 10 · p. 18, §A.9 · p. 18

Related papers

Loading 2610.17709v1…