PaperlayerView exact revision on arXiv

Source-linked AI summary

MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

Yi Qiao, Maolin Zheng, Haoran Sun, Lujia Dong, Wenzhao Lian

arXiv:2610.21686v1cs.RO

TL;DR

Contact-rich anomaly monitoring is often reactive, while visual observations and planned actions alone do not reveal the physical interaction that will follow. MAIA predicts visual and force responses under planned actions, and CRIA provides frame-level anomaly annotations for evaluation. MAIA reports stronger discrimination and earlier warnings on seen and novel tasks, while anomaly-aware action selection raises real-robot success without policy retraining.

  • Problem

    In contact-rich manipulation, visual observations only partly reveal physical interaction and planned actions specify intended motion rather than the environment’s response, while reactive monitoring waits for observable anomalies.

    Source: §I · p. 1

  • Method

    MAIA learns nominal interaction dynamics with a vision–action–force world model that predicts planned actions’ visual and force responses, then uses an anomaly head to predict impending anomalies.

    Source: §I · p. 2

  • Results

    MAIA reports bACC of 0.765 on seen tasks and 0.710 on novel tasks, and raises real-robot success from 52.6% to 61.5% without policy retraining.

    Source: §I · p. 2

  • Takeaways & Limitations

    Predicted anomaly probabilities can guide action selection, improving real-robot task success without retraining the manipulation policy.

    Source: §VII · p. 8

    The reported evaluations cover seen and novel contact-rich tasks and real-robot action selection; the supplied conclusion gives no broader evaluation scope.

    Source: §VII · pp. 7–8

Abstract

from arXiv · show

Reliable contact-rich manipulation requires detecting execution deviations early enough for corrective action. Yet most monitoring methods are reactive, identifying anomalies only after they become observable. Anticipating such anomalies is challenging because visual observations only partially reveal physical interaction, while planned actions describe intended motion rather than their actual physical consequences. We address this problem with MAIA, a multimodal framework for anomaly anticipation. Its vision-action-force world model learns nominal interaction dynamics and predicts the visual and force responses of planned actions, while a lightweight anomaly head predicts impending interaction anomalies. To enable systematic evaluation, we introduce CRIA, a real-world multimodal dataset of 2,852 trajectories spanning 18 contact-rich manipulation tasks, with frame-level annotations of anomaly onset and evolution. Experiments on both seen and novel tasks show that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings, with bACC scores of 0.765 and 0.710 and median anticipation lead times of 28.0 and 25.1 steps, respectively. Moreover, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. These results highlight the value of modeling future physical consequences for proactive anomaly anticipation.

I. INTRODUCTION

Reactive monitoring detects deviations after observable evidence emerges, but contact-rich anomalies can develop without clear task-level failure signals. MAIA anticipates anomalies by predicting planned actions’ visual and force consequences; CRIA supplies temporally localized supervision, and the paper reports improved discrimination and real-robot success.

I. INTRODUCTION

  • Reactive monitoring identifies deviations only after observable evidence emerges, leaving little intervention time when anomalies evolve without clear task-level failure signals.Contact-rich interactions can produce substantially different outcomes under similar actions as the interaction unfolds.Source: §I · p. 1
  • MAIA’s vision–action–force world model learns nominal interaction dynamics and predicts planned actions’ future visual and force responses for a lightweight anomaly head.The head estimates whether an anomaly will occur and its temporal prevalence over the future horizon.Source: §I · p. 2
  • CRIA provides 2,852 real-world trajectories across 18 contact-rich tasks, with temporally aligned visual, action, and force/torque signals and frame-level anomaly-onset and evolution annotations.Such temporally localized supervision is absent from conventional failure-oriented datasets.Source: §I · p. 2
  • MAIA learns interaction dynamics rather than task-specific failure templates and evaluates candidate action sequences before execution, supporting transfer to novel tasks and proactive action selection.The paper reports that anomaly-aware selection improves real-robot success without retraining the manipulation policy.Source: §I · p. 2
  • MAIA achieves bACC of 0.765 on seen tasks and 0.710 on novel tasks, and improves real-robot success from 52.6% to 61.5% without policy retraining.The framework formulates anticipation as prospective action evaluation and predicts future visual and force responses under planned actions.Source: §I · p. 2

II. RELATED WORK

Prior monitoring and world-model approaches provide evidence about execution or imagined futures, but rarely explicitly model physical interaction dynamics under future actions. The paper frames anomalies as local deviations from expected interaction dynamics and predicts whether planned actions will lead to them.

A. Robot Failure and Anomaly Detection

B. World Models for Robot Interaction

  • World models use predicted future representations to support planning and control, while multimodal models add force or tactile sensing; the paper focuses on forecasting visual and force responses conditioned on planned actions.This targets the physical consequences of planned actions rather than only future scenes or next-action generation.Source: B. World Models for Robot Interaction · p. 2

III. PROBLEM FORMULATION

  • The paper defines a manipulation anomaly as an observable deviation from task-oriented expected interaction dynamics, distinct from episode-level task failure.The anticipation target is whether the robot will enter an anomalous state under the policy’s future actions.Source: §III · p. 2
  • Given visual and force history, a future action sequence, and a language task description, the formulation predicts whether any anomaly occurs and its temporal prevalence over the next K steps.The accompanying overview organizes CRIA’s 18 tasks by interaction category and illustrates typical anomaly modes and collection hardware.Source: §III · pp. 2–3, Figure 3 · p. 3

IV. DATASET

CRIA addresses the lack of datasets combining multimodal interaction signals with temporally localized anomaly supervision. It covers contact-rich tasks with synchronized sensing and frame-level labels that track local deviations rather than only episode outcomes.

IV. DATASET

  • CRIA contains 2,852 trajectories across 18 tasks and five categories; 1,817 trajectories include at least one annotated anomalous event.It includes synchronized visual, robot state/action, and force/torque observations, with annotations localizing event emergence, duration, and potential recovery.Source: §IV · p. 3

A. Task Taxonomy

  • The 18 tasks span insertion, pressing, wiping, path-constrained manipulation, and articulated-object manipulation, with category-specific contact patterns and shared interaction dynamics within categories.Tasks within each category vary in object geometry and task semantics, while recurring anomaly patterns appear across categories.Source: A. Task Taxonomy · p. 3

B. Data Collection

  • For each task, collection includes at least 50 successful human demonstrations, 50 anomalous human demonstrations, and 50 autonomous policy rollouts.The three sources cover nominal variability, severe or persistent anomalies, and the policy’s autonomous interaction distribution.Source: B. Data Collection · p. 3

C. Multimodal Recording and Annotation

  • Each trajectory records three aligned RGB views and robot states/actions at 30 Hz, while wrist force/torque is sampled at 120 Hz.Frame labels identify deviations using synchronized visual and physical signals; they reflect interaction process, so successful trajectories may include transient anomalies and failed ones may remain nominal before an anomaly emerges.Source: C. Multimodal Recording and Annotation · pp. 3–4
  • Task-relevant interaction onset uses category-specific criteria rather than a uniform contact threshold.Insertion, pressing, and wiping onset is marked at target vicinity before contact; articulated-object and path-constrained onset follows establishment of the relevant grasp or contact.Source: C. Multimodal Recording and Annotation · p. 4

V. METHOD

MAIA models how interactions evolve under planned actions using a vision–action–force world model, then estimates future anomaly occurrence and prevalence. The model couples action-aligned force context to a pretrained video backbone and predicts both future visual and force responses.

V. METHOD

  • MAIA conditions a pretrained video world model on recent visual and force observations, task instruction, and planned actions, then uses an anomaly head to estimate future anomaly occurrence and temporal prevalence.Training proceeds through nominal dynamics learning, predictive anomaly-head training, and limited-data adaptation to novel tasks.Source: §V · p. 4

A. Vision-Action-Force World Model

  • The VAF world model encodes future actions and recent force history into action–force context tokens, which condition selected video-transformer blocks through gated cross-attention.An interaction readout links those context tokens to predicted visual evolution, and a temporal force decoder predicts future force; the prediction horizon is K = 50 control steps.Source: A. Vision-Action-Force World Model · p. 4
  • The first training stage uses nominal executions to learn expected interaction dynamics, retaining the video flow-matching objective while jointly supervising future-force prediction.The stated combined loss weights video and force losses; the force loss includes future-force reconstruction, temporal-difference, and smoothness terms.Source: A. Vision-Action-Force World Model · p. 4

B. Predictive Anomaly Head

The predictive anomaly head fuses VAF-WM forecasts with interaction features to estimate anomalies across a future window. For limited-data novel-task adaptation, the video backbone is frozen while the action-force pathway and anomaly head are fine-tuned.

B. Predictive Anomaly Head

  • The anomaly head aligns sparse visual predictions with K-step interaction and force features, then uses a temporal transformer to predict window-level anomaly probability, prevalence, and step-level anomalies.A global summary of force history provides auxiliary conditioning.Source: B. Predictive Anomaly Head · pp. 4–5
  • Stage two freezes VAF-WM and generates predictive rollouts from nominal and anomalous executions to train the anomaly head on deployment-aligned representations.This separates anomaly supervision from nominal-dynamics learning.Source: B. Predictive Anomaly Head · p. 5
  • The anomaly objective combines window- and step-level binary cross-entropy losses with a smooth L1 loss for prevalence.Source: B. Predictive Anomaly Head · p. 5

C. Limited-Data Novel-Task Adaptation

  • For novel-task adaptation, the video backbone is frozen while the action-force pathway and anomaly head are jointly fine-tuned.The adaptation objective adds a force-prediction loss computed exclusively on nominal executions, anchoring the adapted pathway to nominal interaction dynamics while allowing task-specific anomaly adaptation.Source: C. Limited-Data Novel-Task Adaptation · p. 5

VI. EXPERIMENTS

The experiments evaluate anomaly monitoring on 13 seen tasks and five held-out novel tasks, using multiple baseline paradigms and window- and event-level metrics. The qualitative comparison reports fewer false alarms on nominal executions and earlier sustained warnings than WCM on anomalous executions.

1) Task Splits and Temporal Sampling:

  • The evaluation holds out five tasks for novel-task testing and uses 10 training trajectories per novel task; streams are aligned at 30 Hz with a 50-step prediction horizon.The main training split uses 13 seen tasks; windows are extracted every 10 steps.Source: 1) Task Splits and Temporal Sampling · p. 5

2) Baselines:

  • Baselines cover OOD detection, VLA-assisted failure detection, reward models, and world-model-based monitoring, with task-specific fitting or calibration restricted to training and validation splits.Source: 2) Baselines · p. 5

3) Metrics:

  • In the qualitative comparison, MAIA produces fewer false alarms on nominal executions and earlier sustained warnings than WCM on anomalous executions.Reward-model warnings are intermittent.Source: Figure 5 · p. 6
  • Window-level evaluation reports AUPRC and bACC, while event-level evaluation uses anticipation rate and median anticipation lead time; Table II covers seen and novel tasks macro-averaged across tasks.Anticipation rate requires a positive-prediction sequence to begin within 50 steps before onset and persist until onset.Source: 3) Metrics · pp. 5–6, Table II · p. 6

B. Comparative Evaluation

MAIA models planned actions’ visual and force consequences to anticipate anomalies, outperforming baselines on seen and novel tasks. Ablations show that force feedback especially helps distinguish interactions that vision and action alone leave ambiguous.

B. Comparative Evaluation

  • MAIA achieves the strongest overall offline anomaly-prediction performance on both seen and novel task splits.The comparison evaluates anomaly discrimination and anticipation; MAIA also estimates temporal prevalence.Source: B. Comparative Evaluation · p. 6
  • SAFE-LSTM’s bACC falls from 0.724 to 0.495 under zero-shot transfer, while MAIA uses predicted visual and force responses for physically grounded anomaly discrimination.OOD-based methods may flag benign execution variations; MAIA produces fewer false alarms in a representative nominal execution.Source: B. Comparative Evaluation · p. 6
  • MAIA’s full-horizon predictions balance anomaly discrimination and anticipation better than reward models or one-step world models.Reward models can anticipate from degraded progress but discriminate weakly; WCM’s one-step objective limits temporal reach. MAIA’s prevalence MAE is 0.18 on seen tasks and 0.28 on novel tasks.Source: B. Comparative Evaluation · p. 6

C. Ablation Studies

  • Fig. 6 illustrates representative cases where physical feedback distinguishes anomalous interactions ambiguous from visual and action cues alone.Insertion shows a complementary pattern: action information is useful, with force further separating plausible motions from invalid physical contact.Source: Figure 6 · p. 7, C. Ablation Studies · p. 7
  • Adding force feedback improves bACC over MAIA-VA by 0.139 in Wiping and 0.094 in Path-Constrained manipulation.Action conditioning alone does not consistently help in these regimes; force predictions reveal subtle changes such as lost contact or unexpected resistance.Source: C. Ablation Studies · p. 7

D. Novel-Task Adaptation

On novel tasks, joint adaptation rapidly improves anomaly discrimination with limited task-specific data. In real-robot trials, anomaly-aware action selection raises success from 52.6% to 61.5% without policy retraining; the framework’s forecasting and evaluation scope remain bounded.

D. Novel-Task Adaptation

  • Novel-task bACC reaches 0.762 after joint adaptation with 25 trajectories, compared with 0.615 without task-specific tuning.Joint adaptation largely saturates after 25 trajectories; head-only adaptation is less data-efficient and performs worse in the low-data regime.Source: D. Novel-Task Adaptation · p. 7

E. Real-Robot Policy Integration

  • 52.6% to 61.5%: anomaly-aware action selection improves overall real-robot task success without retraining the manipulation policy.At each planning step, MAIA scores eight candidate action chunks and executes the first 20 actions of the lowest-anomaly chunk; gains are largest in Wiping and Path-Constrained tasks.Source: E. Real-Robot Policy Integration · p. 7

VII. CONCLUSION AND LIMITATIONS

  • MAIA models visual and force consequences of planned actions to anticipate anomalies before onset, with improved discrimination and earlier warnings on seen and novel tasks.Predicted anomaly probabilities can guide action selection and improve real-robot success without policy retraining.Source: §VII · pp. 7–8
  • Finite-horizon forecasting and fixed-length history may limit reliability for longer horizons or anomalies requiring extended interaction context.Experiments use one robot platform and fixed force/torque sensing setup; cross-embodiment and cross-sensor generalization remain unexplored, and the framework does not address root-cause diagnosis or autonomous recovery.Source: §VII · p. 8

APPENDIX

CRIA spans 18 contact-rich tasks and records temporally distinct anomalous events. Its five categories organize tasks by shared interaction constraints and recurring anomaly patterns, rather than task semantics alone.

1) Dataset Statistics:

  • CRIA contains 2,852 trajectories across 18 tasks and five contact-rich manipulation categories.It records 2,328 anomaly events across 1,817 anomalous trajectories, so some trajectories contain multiple temporally distinct anomalous segments.Source: 1) Dataset Statistics · p. 10

2) Task Categories and Anomaly Modes:

  • The five categories are insertion, wiping, pressing, path-constrained manipulation, and articulated-object manipulation, organized by shared physical interaction regimes.Tasks within a category may differ in objects and goals while sharing contact constraints and recurring anomaly modes.Source: 2) Task Categories and Anomaly Modes · p. 10
  • Insertion anomalies include misalignment, tilting, partial insertion, excessive force, jamming, and repeated retract-and-retry behavior.Successful insertion requires precise alignment and motion along a feasible mating direction.Source: 2) Task Categories and Anomaly Modes · p. 10
  • Wiping anomalies include lost contact, uneven pressure, trajectory deviation, and incomplete coverage; pressing anomalies include insufficient force, misaligned or redundant contact, and incomplete actuation.Wiping requires sustained normal contact during tangential motion, while pressing depends on correctly positioned and directed force.Source: 2) Task Categories and Anomaly Modes · p. 10
  • Path-constrained tasks can show trajectory deviation or slipping, while articulated-object tasks can show missed engagement, jamming, or prolonged pauses from misaligned grasping.These tasks require motion that respects a surrounding geometric path or the object’s joint structure.Source: 2) Task Categories and Anomaly Modes · p. 10

B. Model

MAIA encodes planned actions and wrench history into temporal and global contact representations, then predicts future interaction signals and anomaly measures. An auxiliary relational loss preserves pairwise semantic structure between interaction and task-language features.

1) Architecture:

  • A two-layer MLP embeds future actions, while a TCN encodes wrench history, first-order differences, and magnitudes into temporal contact features and a global contact-state token.These representations provide temporal feedback and global modulation for the action–force context builder.Source: 1) Architecture · p. 10
  • A temporal transformer and MLP decode interaction tokens into future wrench predictions, which are fused with predicted visual latents for anomaly and prevalence estimates.The window classifier aggregates the top 20% of temporal features ranked by auxiliary step-head scores; the prevalence head aggregates all temporal features.Source: 1) Architecture · p. 10
  • The auxiliary relational loss matches pairwise similarities between pooled interaction and task-language features without forcing identical cross-modal representations.It is used during nominal world-model training on samples with retained language conditioning, with weight 0.05.Source: 1) Architecture · p. 12

2) Training:

Training uses separate optimization stages for the world model, anomaly head, and novel-task adaptation. Supplementary results provide task-level anticipation performance and real-robot success-rate breakdowns.

2) Training:

  • All stages use AdamW, bfloat16 mixed precision, gradient clipping at maximum norm 1.0, and a constant learning-rate schedule with linear warmup.The reported runs use four NVIDIA A100 GPUs; action and force/torque signals are normalized with training-split statistics.Source: 2) Training · p. 12
  • VAF-WM training jointly optimizes the video transformer and action-force pathway while freezing the VAE and text encoder; anomaly-head training freezes VAF-WM.Rollout features use a single inference step, matching deployment; the anomaly head uses positive class weight 5.0 for class imbalance.Source: 2) Training · p. 12
  • Novel-task adaptation freezes the video backbone and jointly fine-tunes the action-force pathway and anomaly head.Nominal force regularization is applied only to nominal adaptation windows.Source: 2) Training · p. 12

C. Supplementary Experimental Results

Related papers

Loading 2610.21686v1…