PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder

Daehyung Park, Yuuna Hoshi, Charles C. Kemp

arXiv:1711.00614v1cs.ROcs.LG

TL;DR

Robot-assisted manipulation needs anomaly detection to reduce potential hazards, but combining heterogeneous high-dimensional sensory modalities is difficult. The paper introduces an LSTM-VAE detector that reconstructs expected multimodal distributions and uses reconstruction scores with state-based thresholding. Across robot-assisted feeding evaluations, it outperformed five literature baselines and performed better with 17 raw sensory signals than with four hand-engineered features.

  • Problem

    Robot-assisted feeding requires anomaly detection for potential hazard reduction, while high-dimensional heterogeneous multimodal signal fusion is challenging and feature selection may discard relevant information.

  • Method

    An LSTM-VAE models multimodal observations and temporal dependencies, reconstructs their expected distribution, and detects anomalies using reconstruction likelihood with state-based thresholding.

  • Results

    The detector outperformed five baseline methods, with a 0.044 higher AUC than HMM-GP and a 0.064 higher AUC using 17 sensory signals than using 4 hand-engineered features.

  • Takeaways & Limitations

    The LSTM-VAE detector used high-dimensional multimodal inputs without significant feature-engineering effort and supported online anomaly detection with state-dependent decision boundaries.

Abstract

from arXiv · show

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding executions including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve (AUC) of 0.8710 than 5 other baseline detectors from the literature. We also show the multimodal fusion through the LSTM-VAE is effective by comparing our detector with 17 raw sensory signals versus 4 hand-engineered features.

I. INTRODUCTION

Robot-assisted feeding can benefit from anomaly detection because task complexity, variability, and sensor uncertainty create potential failures, while multimodal fusion offers information for recognizing diverse anomalies. The paper introduces an LSTM-VAE detector and evaluates it on robot-assisted feeding executions.

  • Anomaly detection can help reduce hazards by recognizing highly unusual robot-assisted feeding situations and stopping in those situations.
  • Multimodal sensory signals may detect diverse anomalies, but fusing high-dimensional heterogeneous inputs remains challenging.
  • The proposed LSTM-VAE encodes multimodal observations and temporal dependencies, then reconstructs their expected input distribution.
  • The detector uses reconstruction log-likelihood and a state-based threshold designed to increase sensitivity and lower false alarms.
  • The evaluation used 1,555 robot-assisted feeding executions collected from 24 able-bodied participants.
  • The method achieved a 0.044 higher AUC than HMM-GP and a 0.064 higher AUC with 17 sensory signals than with 4 hand-engineered features.

III. LSTM-BASED VARIATIONAL AUTOENCODING

Autoencoders learn compressed representations and reconstruct inputs, while VAEs additionally model observation distributions through latent variables and variational inference. The VAE objective combines latent-space regularization with reconstruction likelihood.

  • Autoencoder: An autoencoder uses sequential encoder and decoder networks to compress an input and reconstruct it from the resulting latent representation.Training minimizes the difference between the input and reconstructed input.
  • Variational autoencoder: A VAE extends an autoencoder with latent random variables to model the underlying distribution of observations and generate data.Its formulation uses a prior distribution and a likelihood for observations.
  • Variational autoencoder: The VAE maximizes a variational lower bound on log likelihood while optimizing encoder and decoder parameters.The encoder parameters are φ and the decoder parameters are θ.
  • Variational autoencoder: The objective regularizes the latent variable through KL divergence and reconstructs x by maximizing log-likelihood from samples of qφ(z|x).These are the regularization and reconstruction components of the VAE objective.
  • Detection pipeline: The detector architecture trains the LSTM-VAE and a threshold estimator, then flags an anomaly when the current anomaly score exceeds the estimated threshold η.Testing uses sensory signals only; the red training arrows are not used during testing.
  • Variational autoencoder: For continuous data, the VAE commonly uses Gaussian posterior and likelihood distributions, with a standard normal prior as a typical choice.Binary data may instead use a Bernoulli likelihood.

C. An LSTM-based Variational Autoencoder (LSTM-VAE)

The LSTM-VAE combines recurrent temporal modeling with variational encoding and decoding for multimodal time-series inputs. It uses denoising training and a progress-based prior whose center changes across task execution.

  • LSTM-VAE architecture: The LSTM-VAE replaces feed-forward VAE networks with LSTM-based encoder and decoder modules to model temporal dependencies in multimodal inputs.The encoder estimates latent mean and covariance from LSTM outputs, while the decoder reconstructs inputs from latent representations.
  • Denoising training: Denoising variational autoencoding corrupts inputs with Gaussian noise before estimating the latent posterior.The denoising lower bound replaces the standard variational lower bound during training.
  • Denoising training: For computational convenience, the corrupted-input posterior is approximated with a single Gaussian q̃φ(z|x) ≈ qφ(z|x̃).The underlying corrupted-input posterior can be represented as a mixture of Gaussians.
  • Progress-based prior: The progress-based prior varies the center of an isotropic normal distribution from p1 at initial progress to pT at final progress.Its covariance matrix is simplified to Σp = I.
  • Progress-based prior: The varying prior introduces task-execution temporal dependency into the underlying distribution by gradually changing the prior center.This differs from conventional static normal priors and other cited recurrent or transition priors.
  • Implementation: The implementation uses stateful LSTMs with tanh activations, Adam optimization, three-dimensional latent variables, and a 0.001 learning rate.The authors did not use a sliding window, although they note one could be applied.

IV. ANOMALY DETECTION

The paper introduces an online anomaly-detection framework for multimodal sensory signals that uses state-based thresholding. This framework is presented as the paper’s detection approach.

  • The proposed framework performs online anomaly detection on multimodal sensory signals using state-based thresholding.

A. Anomaly Score

The detector scores observations by their negative log-likelihood under an LSTM-VAE reconstruction and compares that score with a state-dependent threshold. Its training framework learns the LSTM-VAE, expected anomaly scores, and online decision process from non-anomalous multimodal sequences.

  • Detection rule: An anomaly is detected when the current anomaly score exceeds threshold η.The online detector processes multimodal input and compares its score with η.
  • Anomaly score: The anomaly score is the negative log-likelihood of an observation under the LSTM-VAE’s reconstructed distribution.A high score indicates poor reconstruction relative to non-anomalous training data.
  • State-based thresholding: The state-based threshold maps latent states to expected anomaly scores and adds a constant c for sensitivity control.An SVR with an RBF kernel estimates the expected score from latent representations.
  • Training framework: Training preprocesses multimodal sequences, optimizes the LSTM-VAE on non-anomalous data, and trains an expected-score estimator using validation representations.Sequences are resampled to length T and modalities are normalized to [0, 1].
  • Online testing: During online testing, the detector scales each input, estimates its latent state and reconstructed distribution, and outputs anomaly or non-anomaly.The LSTM state is reset and updated through the testing process.

A. Instrumental Setup

The experiments used a PR2 mobile manipulator equipped with sensing for visual, auditory, force, torque, position, and current information during feeding. Low-gain control and a 50 Hz model predictive controller were used without haptic feedback.

  • Robot platform: The platform was a Willow Garage PR2 with two 7-DOF arms, powered grippers, and an omni-directional mobile base.The robot provided the general-purpose manipulation platform for robot-assisted feeding.
  • Sensors: Sensors included an RGB-D camera with microphone, a utensil-handle force/torque sensor, joint encoders, and current sensors.These sensors measured mouth position, sound, utensil force, and robot state information.
  • Control: The system used low-gain PID control and a 50 Hz mid-level model predictive controller without haptic feedback.The control configuration was selected for safety and hazard prevention.

B. Data Collection

The study analyzed 1,555 feeding executions from 24 able-bodied participants, combining a prior yogurt-and-silicone-spoon dataset with newly collected non-anomalous data using varied foods and utensils.

  • Participants and executions: 1,555 feeding executions came from 24 able-bodied participants, including 1,203 newly collected non-anomalous executions.The participants were 16 male and 8 female, aged 19–35.
  • Training/testing dataset: The training/testing dataset contained 352 executions, including 160 anomalous and 192 non-anomalous executions from 8 participants.Participants used yogurt and a silicone spoon in this dataset.
  • Pre-training dataset: The pre-training dataset contained 1,203 non-anomalous executions using multiple foods and utensils to initialize LSTM-VAE weights before fine-tuning.It included data from 16 newly recruited participants.
  • Materials: The experimental materials included yogurt, cottage cheese, watermelon chunks, fruit mix, rice, cereal, and five utensils.The training/testing subset specifically used yogurt and a silicone spoon.

C. Experimental Procedure

Participants performed commanded feeding subtasks under constrained body movement while users, experimenters, or the system generated 12 representative anomalies. The study collected and preprocessed multimodal signals for comparison with five baseline detectors.

  • Experimental procedure: A typical sequence involved scooping or stabbing followed by feeding, with commands issued through a web-based interface.Participants were instructed not to move their upper bodies and to eat from the utensil with their lips.
  • Anomaly conditions: The study defined 12 anomalies caused by the user, environment, or system, including collisions, misses, occlusion, noise, and system freeze.Anomalies were presented in randomly ordered anomalous and non-anomalous executions.
  • Signal collection: Each execution provided 17 sensory signals from five sensors, covering sound energy, force, joint torque, spoon position, and mouth position.The signals were organized as sequences with 17 dimensions per time step.
  • Preprocessing: Signals were resampled to 20 Hz, scaled to [0, 1], and converted into sequence-length-by-17 tuples; four hand-engineered features were also extracted.The 20 Hz rate matched the robot’s actual anomaly-check frequency.
  • Baseline methods: Five baselines were implemented: RANDOM, OSVM, HMM-GP, AE, and EncDec-AD.The baselines included one-class, likelihood-based, conventional autoencoder, and LSTM autoencoder detectors.
  • Visualization: Figure 6 compares observed and reconstructed feature distributions above with current, expected, and thresholded anomaly scores below.Brown lines mark detection times, while the dashed curve is the state-based threshold.

VI. EVALUATION

The evaluation shows that the LSTM-VAE reconstructs multimodal execution patterns well enough to distinguish anomalous from non-anomalous feeding executions. State-based thresholding improves decision boundaries, while the detector outperforms baselines with both hand-engineered and raw sensory inputs.

  • Anomaly-score behavior: Anomalous executions produced larger and more variable anomaly scores than non-anomalous executions, making the two conditions distinguishable.The score patterns were evaluated over 24 anomalous and 20 non-anomalous executions during leave-one-person-out cross-validation.
  • Thresholding: State-based thresholding provides tighter, time-varying decision boundaries than fixed thresholding and detects anomalies when actual scores exceed expected scores.In the feeding example, the first detection time matched the initial increase of accumulated force.
  • Baseline comparison: The detector outperformed 5 baseline methods with 4 hand-engineered features, achieving an AUC 0.044 higher than HMM-GP.The comparison used leave-one-person-out cross-validation, training on 7 participants and testing on the remaining participant.
  • Multimodal inputs: With 17 sensory signals and additional pre-training data, the method achieved an AUC 0.064 higher than the next-best method, EncDec-AD.This result also exceeded the performance obtained with hand-engineered features.
  • Thresholding: ROC curves showed that state-based thresholding achieved higher true positive rates than fixed thresholding at the same false positive rates.This comparison used 17 sensory signals with the pre-training dataset.

VII. CONCLUSION

The paper concludes that an LSTM-VAE-based detector can model multimodal signal distributions and detect anomalies in robot-assisted feeding. It reports higher AUC than five baselines, stronger performance with 17 raw inputs than with four hand-engineered features, and benefits from state-based decision boundaries.

  • VII. CONCLUSION: The detector models the underlying distribution of multidimensional signals and uses negative log-likelihood as its anomaly score.The LSTM-VAE reconstructs multimodal signals with expected-distribution information.
  • VII. CONCLUSION: The detector achieved higher AUC than 5 baseline methods in robot-assisted feeding.
  • VII. CONCLUSION: Using 17 raw inputs outperformed a detector trained with 4 hand-engineered features without significant feature-engineering effort.
  • VII. CONCLUSION: State-based decision boundaries supported more sensitive anomaly detection with lower false alarms.
Loading 1711.00614v1…