PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive World-Models for Anomaly Recognition (AWARE): A Framework for Robot Dynamics Prediction and Introspection

Luke Beddow, George Mavroghenis, Rodrigo Chacon Quesada, Kamil Dreczkowski, Cong Sun, Jiankai Wang, Oscar Kwong-Fai Pang, Antoine Cully

arXiv:2610.32742v1cs.ROcs.AI

TL;DR

Camera-only monitoring is needed where onboard sensors are absent or access is restricted, but noisy state estimates complicate anomaly detection. AWARE combines latent dynamics estimation with world-model state prediction; on real-world crane trajectories, it improves high-noise anomaly recognition over RWM and identifies degraded motors with 88.0% accuracy.

  • Problem

    Many infrastructure settings lack onboard sensors or restrict system access, while noisy camera-only state estimates degrade anomaly detection and system identification.

    Source: §1 · p. 2

  • Method

    AWARE jointly uses a latent dynamics estimator and a conditioned world model, comparing dynamics and prediction signals with normal-operation references to detect anomalies and identify fault sources.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    26% improvement over RWM in high-noise AP, with 88.0% accuracy identifying the impaired motor under high-noise estimation.

    Source: §5 · pp. 7–8

  • Takeaways & Limitations

    AWARE supports passive camera-only anomaly monitoring and fault localization on the evaluated robotic crane platform.

    Source: §5 · p. 8, §7 · p. 9

    AWARE assumes fixed operating conditions under its reference distribution, so a new non-anomalous operating point may trigger false positives.

    Source: §6 · p. 9

Abstract

from arXiv · show

Unrecognized anomalies in critical infrastructure can rapidly cascade into catastrophic physical or operational failures. Current anomaly detection approaches often rely on direct access to the monitored system by utilizing onboard sensors, such as motor encoders, to detect anomalous states. However, legacy hardware often lacks these sensors, or strict security protocols may prohibit access altogether. We therefore present AWARE, a world model framework that is capable of anomaly detection using only external camera monitoring via system dynamics estimation and state prediction. Evaluated across 240 real-world robotic crane trajectories, including payload collisions and motor degradations, AWARE demonstrates a 26% improvement in anomaly classification over our strongest baseline under noisy, camera-only state estimation. Furthermore, the inferred dynamics parameter can be decoded to identify the source of faults, localizing the degrading motor with 88% accuracy.

1 Introduction

Critical infrastructure needs monitoring for anomalous behavior, but many systems lack onboard sensors or prohibit direct access, making camera-only state estimates noisy. AWARE combines latent dynamics estimation and state prediction to detect anomalies and identify fault sources.

1 Introduction

  • Critical infrastructure disruptions can endanger nearby workers and cause economic disruption, making monitoring for anomalous behavior important for safe, continuous operation.Many applications lack onboard sensors or restrict access, so passive camera-based state estimation introduces noise throughout the detection pipeline.Source: §1 · p. 2
  • AWARE jointly trains a latent dynamics estimator and world model, using domain randomization to address noisy camera-only estimation without perfect system identification or explicit anomaly priors.The latent representation captures intrinsic parameters such as motor gain or mass and conditions the model’s next-state prediction.Source: §1 · p. 2
  • 26% improvement over the next best baseline under high-noise state estimation accompanies 88% identification of in-distribution anomaly sources.Across 240 real-world robotic crane trajectories, AWARE also showed a 3% improvement under low-noise state estimation.Source: §1 · p. 2

2 Related Work

Prior anomaly methods address unexpected observations or temporal deviations but often depend on onboard sensing, labeled anomalies, or semantic scene understanding. AWARE combines a dynamics-aware world model with latent parameter estimation for passive monitoring without onboard sensing.

2 Related Work

  • Statistical and reconstruction approaches score observations against a reference distribution, while recent spatiotemporal methods often assume labeled anomaly data unavailable in semi-supervised infrastructure settings.Foundation-model approaches offer semantic scene understanding but may miss subtle changes in system dynamics.Source: §2 · pp. 2–3
  • Prior predictive temporal models learn nominal dynamics but assume dense onboard sensing; AWARE instead targets passive monitoring with no onboard sensing.The framework combines dynamics-aware prediction with latent parameter estimation to detect anomalies that may be visually subtle.Source: §2 · p. 3
  • AWARE adds latent dynamics estimation to world models, supplying an explicit dynamics prior and a second anomaly signal beyond state prediction or reconstruction.The latent deviation is intended to improve robustness, while a decoder supports introspection into dynamics parameters.Source: §2 · p. 3

3 Adaptive World-Models for Anomaly Recognition (AWARE)

AWARE estimates latent dynamics from state-action history and conditions a world model to predict the next state. It compares both signals with nominal reference distributions to detect changes in modeled and unmodeled dynamics.

3 Adaptive World-Models for Anomaly Recognition (AWARE)

  • AWARE combines a latent dynamics estimate with next-state prediction, comparing both against normal-operation distributions for anomaly detection and introspection.The latent estimator processes state-action history; the world model uses that representation, the same history, and the current action to predict the next state.Source: §3 · p. 3

3.1 Problem Formulation

  • Anomalies are defined as changes from the initial dynamics context, with latent estimates tracking modeled changes and prediction errors detecting unmodeled changes.The context comprises modeled parameters γ and unmodeled dynamics ζ.Source: §3.1 · p. 4

3.2 Dynamics Parameter Estimation

  • A student estimator learns from state-action history to predict a latent representation of randomized dynamics parameters, using a teacher encoder shaped by the downstream world model.A System Parameter Decoder recovers γ from the latent representation for introspection; compression and regularization are removed to preserve information.Source: §3.2 · p. 4

3.3 World Model State Prediction

  • The world model approximates the transition function by outputting a Gaussian distribution over the next state conditioned on history, current action, and latent dynamics.The paper evaluates Transformer and RSSM backbones.Source: §3.3 · p. 4

3.4 Anomaly Decision

  • At test time, Mahalanobis distances from nominal reference distributions yield separate latent and prediction-error anomaly scores for modeled and unmodeled changes.Prediction error is averaged over a window and normalized by predicted uncertainty; thresholds are calibrated during nominal operation.Source: §3.4 · pp. 4–5

4 Experimental Evaluation

The evaluation uses a simulated, domain-randomized crane model to train and test anomaly detectors on human-operated crane trajectories. It compares high-quality motion-capture estimates with noisier camera-only estimates and evaluates both classification performance and model baselines.

Experimental Procedure and Task Design

  • Training data came from a MuJoCo crane simulation calibrated by grid search, then generated with an exploratory policy and domain randomization around ground-truth parameters.The randomization removes the requirement for perfect system identification.Source: §4 · p. 5
  • 240 human-in-the-loop trajectories comprise 80 nominal runs and 160 anomalous runs, including payload collisions and motor degradations.Five operators moved the payload through three randomly sampled waypoints; the dataset was screened to exclude perception failures.Source: §4 · p. 5

State Estimation

  • The camera-only pipeline estimates crane joints and operator actions using cameras, ArUco markers, a CNN, and hand tracking, while the comparison pipeline uses motion capture and proprioception.Camera-only estimates are substantially noisier and represent passive monitoring without proprioceptive sensors.Source: §4 · p. 6, Figure 3 · p. 6

Metrics

  • Performance is measured by trajectory-level Average Precision and F1 across 240 trajectories, with repeated evaluations and cross-validation for threshold selection.The procedure uses 10 repeats and a stratified 5-fold cross-validation assessment of generalization.Source: §4 · p. 6

Model Baselines

  • The baseline set includes standalone Transformer and RSSM world models, non-temporal anomaly detectors, foundation-model methods, ROA, and RWM.AnyAnomaly is evaluated only on nominal and payload-collision trajectories.Source: §4 · pp. 6–7, Table 1 · p. 7

5 Results and Discussion

AWARE’s Transformer model outperforms the strongest baseline across both state-estimation conditions, with a larger advantage under camera-only noise. Its latent and prediction-error signals contribute differently across anomaly types, while the dynamics latent also supports motor-strength estimation and fault identification.

Anomaly Recognition

  • 0.93 AP (low noise) and 0.77 AP (high noise) exceed RWM by 3% and 26%, respectively, with statistically significant improvements in AP and F1 across both noise conditions.State-prediction-error-only methods suffer an average 44% AP degradation under high noise.Source: §5 · p. 7
  • The latent signal mostly detects in-distribution motor degradations, while prediction error detects more severe degradation and payload anomalies.The results support the interpretation that the latent signal detects changes in modeled dynamics and prediction error detects unmodeled dynamics; this distinction is more pronounced at low noise.Source: §5 · pp. 7–8, Figure 4 · p. 8

System Introspection

  • 18.1% MAE and 88.0% impaired-motor identification accuracy under high noise compare with 16.7% MAE and 93.3% accuracy under low noise.The estimates are aggregated across motor-degradation conditions and averaged over the final 100 trajectory timesteps.Source: §5 · p. 8, Figure 5 · p. 8

Discussion

  • The dual-signal architecture combines prediction error for unmodeled anomalies with latent deviation for modeled anomalies, while latent deviation is more noise-resilient than prediction error.Prediction error can be made more sensitive to modeled anomalies when fixed on nominal dynamics.Source: §5 · pp. 8–9
  • Foundation-model baselines perform worse than dynamics-based methods across both noise levels, while the AWARE dynamics representation outperforms the control-informed ROA representation and RWM’s raw parameter prediction.AESOP and AnyAnomaly are the foundation-model methods discussed.Source: §5 · p. 9

System Introspection

  • Motor-strength estimates stabilize about 80 timesteps after anomaly onset, and in-distribution degradations are decoded more accurately than out-of-distribution cases.The paper relates the accuracy gap to the limits of domain randomization.Source: Figure 5 · p. 8, §5 · p. 9

6 Limitations

AWARE assumes operating conditions remain fixed relative to its reference distribution, and its demonstrated generalization is limited to one platform. Figure 6 also estimates camera-only noise relative to motion-capture quality.

6 Limitations

  • Figure 6 estimates camera-only noise relative to motion-capture quality by inverse-interpolating the Average Precision score.The cross marks the interpolated noise level, scaled to each joint’s standard deviation.Source: Figure 6 · p. 9
  • AWARE assumes fixed operating conditions relative to its reference distribution; a new non-anomalous operating point may trigger false positives.The authors suggest periodic reference updates or separate intrinsic and extrinsic reference distributions as possible future work.Source: §6 · p. 9
  • Evaluation on a single platform limits the generalization AWARE has demonstrated so far.The authors identify validation on other systems as future work.Source: §6 · p. 9

7 Conclusions

AWARE combines latent dynamics estimation with state prediction for camera-only anomaly monitoring, reporting detection and fault-localization results on a robotic crane. The appendix describes the crane’s joints, modeled dynamics parameters, and camera-based hardware setup.

7 Conclusions

  • 93% AP with low-noise data and 77% AP under camera-only estimation accompany a 26% improvement over baselines; faulty-motor isolation reaches 88%.These results are reported on a real-world robotic crane dataset of 240 trajectories.Source: §7 · p. 9

A.1 Robotic Platform

  • The under-actuated crane has three actuated joints—slew, luff, and hoist—and four free joints modeling cable and payload motion.The free joints capture pendulum swing and double-pendulum effects when the payload is not aligned with the cable.Source: §A.1 · p. 13
  • The hidden dynamics parameter γ has seven components for motor inertial and maximum-velocity properties plus payload mass.The study focuses on motor properties and compares decoded motor strength with ground truth.Source: §A.1 · p. 13
  • Two Stereolabs cameras feed state estimation on a Jetson AGX Orin, while a Raspberry Pi 4 interfaces with the crane’s control bus.The control architecture and hardware interface use ROS 2 Humble.Source: §A.1 · p. 13

A.2 Model Training and Hyperparameters

Training uses MuJoCo crane data, ground-truth parameter calibration, and domain randomization to cover uncertain dynamics and noisy observations. The appendix also specifies model history and prediction horizons and identifies the scopes of Tables 2 and 3.

A.2 Model Training and Hyperparameters

  • Ground-truth dynamics parameters γ* were established by fine-tuning γ against real-crane motion-capture data with a grid search over six motor parameters.The models themselves were trained on data generated from a MuJoCo XLA crane simulation.Source: §A.2 · p. 13
  • Domain randomization centered on γ* uses parameter-specific ranges, while payload mass is randomized absolutely.The ranges reflect deployment settings where fine-tuning is unavailable and uncertainty is high; Table 2 summarizes the randomization scales.Source: §A.2 · p. 13, Table 2 · p. 14
  • Simulation training uses exploratory joint-control policies and models sensor noise, including time-correlated noise, repeated readings, offsets, and spikes.The four sub-policies are random, random-walk, sinusoid, and no-motion; γt receives Gaussian noise and changes during trajectories.Source: §A.2 · p. 13
  • Models use 64 history timesteps and predict 20 future timesteps during anomaly detection.Transformer and RSSM world-model backbones have 600,000 and 400,000 parameters, respectively; each AWARE latent dynamics estimator has 500,000.Source: §A.2 · p. 13
  • Table 3 organizes anomaly-recognition results by decision input, with bold entries marking statistically significant gains over the best non-AWARE model.AnyAnomaly is evaluated only on the nominal and payload-collision set.Source: Table 3 · p. 14

B.1 Full Results Table

Combining prediction and dynamics errors gives the strongest anomaly-detection results, while the latent signal is more noise-resilient but insufficient on its own. Adding prediction error improves high-noise performance.

B.1 Full Results Table

  • 18% average AP improvement in the high-noise setting comes from combining prediction error Epred with latent dynamics error Elat.The combined signals achieve the best results among the anomaly decision signals reported in Table 3.Source: §B.1 · p. 14
  • Elat-only methods show an average 17% drop across models when moving from low- to high-noise regimes.The passage describes these methods as having the best noise resilience across that transition.Source: §B.1 · p. 14
  • Elat alone is not sufficient for accurate anomaly detection, despite its greater noise resilience.The passage reports the combined prediction-and-dynamics signal as the best-performing decision input.Source: §B.1 · p. 14

B.2 Statistical Significance Tests

RSSM AWARE was the strongest learned predictor over a full 100-step horizon with noisy inputs, while statistical tests found consistent cross-seed improvements over most baselines but weaker evidence for some camera-only comparisons.

B.2 Statistical Significance Tests

  • For nearly all baseline–metric pairs, Wilcoxon tests reported p = 0.00195, below the Bonferroni-corrected α = 0.0125 threshold across four metrics.Tests compared 10 paired seeds.Source: §B.2 · p. 14, Table 4 · p. 15
  • Evidence was weaker for Transformer AWARE versus RSSM AWARE on high-noise metrics (p = 0.02734 and p = 0.1934), and for RSSM AWARE versus RWM on low-noise F1 (p = 0.86523).The authors also note weaker evidence for RSSM AWARE improvements under camera-only estimation.Source: §B.2 · pp. 14–15

B.3 World Model Prediction Error

  • With noisy inputs, RSSM AWARE outperformed MuJoCo over the full 100-step horizon, and all learned world models were more accurate than MuJoCo over a short horizon.The evaluation compared predictions over 100 steps, equivalent to 5 seconds.Source: §B.3 · p. 15

C.1 AWARE Components

The ablations identify the latent representation and nominal conditioning as important to AWARE’s prediction-based anomaly score, while foundation-model evaluations expose limitations on this crane dataset.

C.1 AWARE Components

  • Removing the latent bottleneck improved performance; encoder regularization had negligible effect, while removing the decoder offered little gain at the cost of introspection.The ablation training curves cover 1.5 million training steps on the Single Camera Quality dataset.Source: §C.1 · pp. 15–16, Figure 8 · p. 17

C.2 Prediction Anomaly Score

  • Using a moving latent instead of the fixed median nominal latent significantly reduced prediction-based anomaly detection, while omitting variance normalization hurt more under high noise.The authors report only marginal reduction from omitting normalization under low noise.Source: §C.2 · p. 16, Table 5 · p. 17

C.3 Foundation Models

  • AESOP underperformed AWARE by 46%; the authors attribute this partly to its lightweight language model and to textual descriptions losing rich semantics from raw images.The text-description approach is described as a reduced representation.Source: §C.3 · pp. 17–18
  • On payload-collision and nominal cases, AnyAnomaly achieved AP of 0.37 and F1 of 0.56, with 21/40 false negatives; motor-degradation evaluation was outside its four-frame scope.The method uses a short sequence of four key frames.Source: §C.3 · p. 18

D.1 Measurement Signal Processing

AWARE’s two measurement signals capture different anomaly patterns: latent deviations are more useful for modeled dynamics changes, while prediction error better detects unmodeled and instantaneous events.

D.1 Measurement Signal Processing

  • The anomaly measurement uses Mahalanobis distance between prediction error and a reference distribution; the authors report that it performed better than hand-tuned normalization.They suggest its full covariance accounting may explain the gain.Source: §D.1 · p. 19
  • Moving from L1/L2 to standardized L2 and Mahalanobis measures significantly improved performance, while the gain from full covariance over diagonal normalization was marginal.The authors attribute the larger improvement to normalizing by the diagonal covariance and reducing feature scale effects.Source: §D.1 · p. 19, Table 6 · p. 19
  • Latent deviation became less reliable as motor strength fell from 25% to 0%, whereas prediction error detected more motor-degradation cases and was more effective for payload anomalies.The authors interpret the signals as responding to modeled dynamics changes (γ) and unmodeled dynamics changes (ζ), respectively; the pattern was stronger at low noise.Source: §D.1 · pp. 19–20
  • A detection is successful when a pivoting measurement signal exceeds its threshold; the example plots distinguish true and false positives and negatives around anomaly injection.The plots show predictor and latent-estimator signals with separate thresholds.Source: Figure 9 · p. 20, Figure 10 · p. 21
Loading 2610.32742v1…